[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:01 dist-test-slave-0b58 krb5kdc[18799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:01 dist-test-slave-0b58 krb5kdc[18799](info): set up 2 sockets
Aug 06 16:12:01 dist-test-slave-0b58 krb5kdc[18799](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 16:12:02.073140 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:02.206108 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44971
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:02.333575 18812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:02.333948 18812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:02.334038 18812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:02.338305 18812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:02.338408 18812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:02.338443 18812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:02.338481 18812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:02.338517 18812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:02.344563 18812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:02.345979 18812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:02.347374 18812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:02.356967 18818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:02.356948 18812 server_base.cc:1034] running on GCE node
W20260806 16:12:02.356940 18819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:02.356930 18821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:02.357899 18812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:02.359086 18812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:02.360245 18812 hybrid_clock.cc:648] HybridClock initialized: now 1786032722360216 us; error 47 us; skew 500 ppm
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:02.368546 18812 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:02.369625 18812 webserver.cc:460] Webserver started at http://127.18.90.62:46459/ using document root <none> and password file <none>
I20260806 16:12:02.370260 18812 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:02.370342 18812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:02.370570 18812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:02.372879 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "702419b196c44889b3e87a73c6b4243e"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "7bcef52593321285baa0a89b4e1fabb7"
server_key_iv: "3b143908f47182bb7e0cbde42b1e2573"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.373484 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "702419b196c44889b3e87a73c6b4243e"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "7bcef52593321285baa0a89b4e1fabb7"
server_key_iv: "3b143908f47182bb7e0cbde42b1e2573"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.377696 18812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 16:12:02.380383 18828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.381769 18812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:02.381898 18812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "702419b196c44889b3e87a73c6b4243e"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "7bcef52593321285baa0a89b4e1fabb7"
server_key_iv: "3b143908f47182bb7e0cbde42b1e2573"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.382012 18812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:02.399487 18812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:02.402824 18812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:02.403050 18812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:02.411294 18812 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:44971
I20260806 16:12:02.411307 18880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:44971 every 8 connection(s)
I20260806 16:12:02.412439 18812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:02.415635 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:02.418838 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 18812
I20260806 16:12:02.419029 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:02.420475 18792 external_mini_cluster.cc:1428] Setting key 51e4df0fb91838af908a82b16435819d
I20260806 16:12:02.421650 18882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e: Bootstrap starting.
I20260806 16:12:02.424237 18882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:02.425122 18882 log.cc:824] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:02.427174 18882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e: No bootstrap required, opened a new log
I20260806 16:12:02.431836 18882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702419b196c44889b3e87a73c6b4243e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44971 } }
I20260806 16:12:02.432087 18882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:02.432147 18882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702419b196c44889b3e87a73c6b4243e, State: Initialized, Role: FOLLOWER
I20260806 16:12:02.432646 18882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702419b196c44889b3e87a73c6b4243e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44971 } }
I20260806 16:12:02.432797 18882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:02.432864 18882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:02.432961 18882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:02.433920 18882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702419b196c44889b3e87a73c6b4243e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44971 } }
I20260806 16:12:02.434262 18882 leader_election.cc:302] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 702419b196c44889b3e87a73c6b4243e; no voters:
I20260806 16:12:02.434587 18882 leader_election.cc:288] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:02.434723 18887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:02.434949 18887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [term 1 LEADER]: Becoming Leader. State: Replica: 702419b196c44889b3e87a73c6b4243e, State: Running, Role: LEADER
I20260806 16:12:02.435294 18887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702419b196c44889b3e87a73c6b4243e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44971 } }
I20260806 16:12:02.435949 18882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:02.438634 18890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 702419b196c44889b3e87a73c6b4243e. Latest consensus state: current_term: 1 leader_uuid: "702419b196c44889b3e87a73c6b4243e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702419b196c44889b3e87a73c6b4243e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44971 } } }
I20260806 16:12:02.438875 18890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:02.439496 18888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "702419b196c44889b3e87a73c6b4243e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702419b196c44889b3e87a73c6b4243e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44971 } } }
I20260806 16:12:02.439677 18888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e [sys.catalog]: This master's current role is: LEADER
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
W20260806 16:12:02.441738 18902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:12:02.441887 18902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:12:02.441964 18904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:02.443148 18904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:02.447888 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:02.424023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.18.90.62:44971)
0806 16:12:02.424648 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:02.424667 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:02.425776 (+ 1109us) server_negotiation.cc:407] Connection header received
0806 16:12:02.429490 (+ 3714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:02.429507 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:02.429821 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:02.430152 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:02.432587 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.433680 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.435358 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.436062 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.436332 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:02.441918 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:02.441925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:02.441943 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:02.441976 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:02.443812 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:02.444527 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:02.444533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:02.444539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:02.444679 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:02.445144 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:02.445148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:02.445150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:02.445658 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:02.445887 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:02.446500 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 16:12:02.446803 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":194,"threads_started":1}
I20260806 16:12:02.449793 18904 catalog_manager.cc:1347] Generated new cluster ID: cfae172761264bcc8d3a35402ff2aaa1
I20260806 16:12:02.449858 18904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:02.463680 18904 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:02.464780 18904 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:02.473639 18904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 702419b196c44889b3e87a73c6b4243e: Generated new TSK 0
I20260806 16:12:02.474484 18904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:02.512782 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44971
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:02.628801 18909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:02.629181 18909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:02.629266 18909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:02.634096 18909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:02.634183 18909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:02.634303 18909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:02.640326 18909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44971
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:02.641535 18909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:02.642623 18909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:12:02.653047 18909 server_base.cc:1034] running on GCE node
W20260806 16:12:02.653007 18917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:02.653028 18915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:02.653007 18914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:02.653745 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:02.654417 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:02.655601 18909 hybrid_clock.cc:648] HybridClock initialized: now 1786032722655555 us; error 58 us; skew 500 ppm
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:02.658496 18909 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:02.659425 18909 webserver.cc:460] Webserver started at http://127.18.90.1:35981/ using document root <none> and password file <none>
I20260806 16:12:02.660032 18909 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:02.660125 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:02.660415 18909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:02.662560 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "c046e7d81a8f422382540cc4d771241a"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "8972c3ce416414b2485343daa4ad5abc"
server_key_iv: "3a2f36a8a7732e68411e0569e31395f4"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.663173 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "c046e7d81a8f422382540cc4d771241a"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "8972c3ce416414b2485343daa4ad5abc"
server_key_iv: "3a2f36a8a7732e68411e0569e31395f4"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.667148 18909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 16:12:02.669768 18925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.671057 18909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:02.671195 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "c046e7d81a8f422382540cc4d771241a"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "8972c3ce416414b2485343daa4ad5abc"
server_key_iv: "3a2f36a8a7732e68411e0569e31395f4"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.671319 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:02.692144 18909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:02.695418 18909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:02.695638 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:02.696332 18909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:02.697196 18909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:02.697263 18909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.697327 18909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:02.697366 18909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.706185 18909 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:43561
I20260806 16:12:02.706209 19039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:43561 every 8 connection(s)
I20260806 16:12:02.707144 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:02.710543 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 18909
I20260806 16:12:02.710669 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:02.710915 18792 external_mini_cluster.cc:1428] Setting key a358e9e46b4e3e98627969f08e877096
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:02.721863 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:02.709175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:57089 (local address 127.18.90.62:44971)
0806 16:12:02.709331 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:02.709334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:02.710396 (+ 1062us) server_negotiation.cc:407] Connection header received
0806 16:12:02.711422 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:02.711426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:02.711486 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:02.711588 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:02.712854 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.713397 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.714560 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.714809 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.714917 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:02.717696 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:02.717703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:02.717708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:02.717742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:02.719524 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:02.720053 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:02.720059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:02.720061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:02.720167 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:02.720587 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:02.720593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:02.720597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:02.720861 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:02.720969 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:02.721519 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 16:12:02.721667 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:12:02.722663 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:02.709574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44971 (local address 127.18.90.1:57089)
0806 16:12:02.710239 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:02.710289 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:02.711216 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:02.711778 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:02.711785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:02.712151 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:02.712704 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:02.712719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.713529 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.713532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:02.714434 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:02.714444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.714962 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.714966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:02.715164 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:02.715743 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:02.715769 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:02.717512 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:02.719649 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:02.719654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:02.719667 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:02.719943 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:02.720343 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:02.720345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:02.720347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:02.720489 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:02.720971 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:02.720977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:02.721234 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 16:12:02.721401 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 16:12:02.721682 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":118,"threads_started":1}
I20260806 16:12:02.724035 19040 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44971
I20260806 16:12:02.724375 19040 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:12:02.725003 19040 heartbeater.cc:511] Master 127.18.90.62:44971 requested a full tablet report, sending...
I20260806 16:12:02.727309 18845 ts_manager.cc:194] Registered new tserver with Master: c046e7d81a8f422382540cc4d771241a (127.18.90.1:43561)
I20260806 16:12:02.729472 18845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:57089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:02.742483 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44971
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:02.859277 19045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:02.859575 19045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:02.859658 19045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:02.863848 19045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:02.863921 19045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:02.863983 19045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:02.869472 19045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44971
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:02.870580 19045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:02.871587 19045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:02.882654 19051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:02.882650 19053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:02.882709 19050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:02.882982 19045 server_base.cc:1034] running on GCE node
I20260806 16:12:02.883445 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:02.884140 19045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:02.885330 19045 hybrid_clock.cc:648] HybridClock initialized: now 1786032722885314 us; error 31 us; skew 500 ppm
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:02.888609 19045 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:02.889576 19045 webserver.cc:460] Webserver started at http://127.18.90.2:46363/ using document root <none> and password file <none>
I20260806 16:12:02.890136 19045 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:02.890201 19045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:02.890372 19045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:02.892601 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "4aec83cc38a34b9eaf0e8e090d33e64d"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "295a86ac022ad05da4e797ff24da28b6"
server_key_iv: "4d54890aa95a944547d2ea3223691efb"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.893143 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "4aec83cc38a34b9eaf0e8e090d33e64d"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "295a86ac022ad05da4e797ff24da28b6"
server_key_iv: "4d54890aa95a944547d2ea3223691efb"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.897154 19045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:02.899758 19060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.901196 19045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:12:02.901353 19045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "4aec83cc38a34b9eaf0e8e090d33e64d"
format_stamp: "Formatted at 2026-08-06 16:12:02 on dist-test-slave-0b58"
server_key: "295a86ac022ad05da4e797ff24da28b6"
server_key_iv: "4d54890aa95a944547d2ea3223691efb"
server_key_version: "encryptionkey@0"
I20260806 16:12:02.901486 19045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:02.947551 19045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:02.951179 19045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:02.951345 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:02.951949 19045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:02.952872 19045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:02.952920 19045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.952962 19045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:02.952978 19045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:02.962121 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:37465
I20260806 16:12:02.962143 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:37465 every 8 connection(s)
I20260806 16:12:02.963050 19045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:02.970520 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19045
I20260806 16:12:02.970647 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:02.970916 18792 external_mini_cluster.cc:1428] Setting key 0370ac862800fa778ecdbdd50ef0029c
Aug 06 16:12:02 dist-test-slave-0b58 krb5kdc[18799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:02.977444 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:02.964891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:40929 (local address 127.18.90.62:44971)
0806 16:12:02.965033 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:02.965037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:02.966097 (+ 1060us) server_negotiation.cc:407] Connection header received
0806 16:12:02.967024 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:02.967027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:02.967089 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:02.967187 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:02.968427 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.968966 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.970128 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.970403 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.970512 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:02.973248 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:02.973263 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:02.973265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:02.973289 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:02.974934 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:02.975626 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:02.975629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:02.975631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:02.975719 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:02.976133 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:02.976135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:02.976136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:02.976377 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:02.976470 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:02.977148 (+ 678us) server_negotiation.cc:299] Negotiation successful
0806 16:12:02.977248 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:12:02.978705 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:02.965293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44971 (local address 127.18.90.2:40929)
0806 16:12:02.965947 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:02.965990 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:02.966812 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:02.967365 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:02.967372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:02.967741 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:02.968229 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:02.968242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.969093 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.969097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:02.969998 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:02.970008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:02.970562 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:02.970567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:02.970817 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:02.971548 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:02.971567 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:02.973083 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:02.975047 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:02.975055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:02.975074 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:02.975511 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:02.975811 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:02.975817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:02.975822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:02.976027 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:02.976494 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:02.976503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:02.976864 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 16:12:02.977120 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 16:12:02.977520 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":101,"threads_started":1}
I20260806 16:12:02.979911 19177 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44971
I20260806 16:12:02.980193 19177 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:02.980880 19177 heartbeater.cc:511] Master 127.18.90.62:44971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:12:02.982084 18845 ts_manager.cc:194] Registered new tserver with Master: 4aec83cc38a34b9eaf0e8e090d33e64d (127.18.90.2:37465)
I20260806 16:12:02.982656 18845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:40929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:02.999133 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44971
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:03.115743 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:03.116055 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:03.116104 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:03.120311 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:03.120390 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:03.120496 19182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:03.126077 19182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44971
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:03.127338 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:03.128512 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:03.139010 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:03.139010 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:03.139014 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:03.139446 19182 server_base.cc:1034] running on GCE node
I20260806 16:12:03.139936 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:03.140728 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:03.141923 19182 hybrid_clock.cc:648] HybridClock initialized: now 1786032723141906 us; error 29 us; skew 500 ppm
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032723, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:03.144876 19182 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:03.145907 19182 webserver.cc:460] Webserver started at http://127.18.90.3:45389/ using document root <none> and password file <none>
I20260806 16:12:03.146528 19182 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:03.146622 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:03.146867 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:03.149137 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "c87c5769bd0b4b1e88146d78340657ca"
format_stamp: "Formatted at 2026-08-06 16:12:03 on dist-test-slave-0b58"
server_key: "5bc9425b8c95ee2f901e379b68e4a7b7"
server_key_iv: "21efed2759f40a4caec2f8c63857a375"
server_key_version: "encryptionkey@0"
I20260806 16:12:03.150041 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "c87c5769bd0b4b1e88146d78340657ca"
format_stamp: "Formatted at 2026-08-06 16:12:03 on dist-test-slave-0b58"
server_key: "5bc9425b8c95ee2f901e379b68e4a7b7"
server_key_iv: "21efed2759f40a4caec2f8c63857a375"
server_key_version: "encryptionkey@0"
I20260806 16:12:03.154227 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 16:12:03.157006 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:03.158432 19182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:03.158602 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "c87c5769bd0b4b1e88146d78340657ca"
format_stamp: "Formatted at 2026-08-06 16:12:03 on dist-test-slave-0b58"
server_key: "5bc9425b8c95ee2f901e379b68e4a7b7"
server_key_iv: "21efed2759f40a4caec2f8c63857a375"
server_key_version: "encryptionkey@0"
I20260806 16:12:03.158739 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:03.179028 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:03.182317 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:03.182538 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:03.183187 19182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:03.184088 19182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:03.184158 19182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:03.184239 19182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:03.184314 19182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:03.194135 19182 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:42231
I20260806 16:12:03.194159 19312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:42231 every 8 connection(s)
I20260806 16:12:03.195161 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:03.196871 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19182
I20260806 16:12:03.197010 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:03.197320 18792 external_mini_cluster.cc:1428] Setting key 71e36871a6bfc405ba341db142ce8d9d
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[18799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032723, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:03.210050 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:03.197337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:60147 (local address 127.18.90.62:44971)
0806 16:12:03.197484 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:03.197487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:03.198842 (+ 1355us) server_negotiation.cc:407] Connection header received
0806 16:12:03.199772 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:03.199776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:03.199837 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:03.199935 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:03.201270 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.201812 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.203026 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.203255 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.203366 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:03.206166 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:03.206173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:03.206178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:03.206215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:03.207699 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:03.208330 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:03.208334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:03.208336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:03.208432 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:03.208801 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:03.208805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:03.208807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:03.209029 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:03.209137 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:03.209704 (+ 567us) server_negotiation.cc:299] Negotiation successful
0806 16:12:03.209865 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:12:03.211082 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:03.197875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44971 (local address 127.18.90.3:60147)
0806 16:12:03.198655 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:03.198727 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:03.199539 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:03.200126 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:03.200134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:03.200599 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:03.201117 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:03.201131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.201949 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.201952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:03.202879 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:03.202889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.203415 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.203419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:03.203605 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:03.204205 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:03.204222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:03.205928 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:03.207823 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:03.207828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:03.207843 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:03.208146 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:03.208533 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:03.208537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:03.208539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:03.208698 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:03.209138 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:03.209147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:03.209443 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 16:12:03.209648 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 16:12:03.210015 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":151,"threads_started":1}
I20260806 16:12:03.212303 19313 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44971
I20260806 16:12:03.212579 19313 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:03.213083 19313 heartbeater.cc:511] Master 127.18.90.62:44971 requested a full tablet report, sending...
I20260806 16:12:03.214082 18845 ts_manager.cc:194] Registered new tserver with Master: c87c5769bd0b4b1e88146d78340657ca (127.18.90.3:42231)
I20260806 16:12:03.214643 18845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:60147
I20260806 16:12:03.224772 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032723, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[18799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032723, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:03.252979 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:03.243259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.18.90.1:43561)
0806 16:12:03.243595 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:03.243602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:03.243672 (+ 70us) server_negotiation.cc:407] Connection header received
0806 16:12:03.243735 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:03.243740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:03.243936 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:03.244058 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:03.244834 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.245609 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.246568 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.246962 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.247045 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:03.249519 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:03.249531 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:03.249545 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:03.249572 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:03.251186 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:03.251605 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:03.251611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:03.251612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:03.251702 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:03.251930 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:03.251932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:03.251933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:03.252186 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:03.252413 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:03.252596 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 16:12:03.252771 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
W20260806 16:12:03.254014 18953 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41938
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[18799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:03.281837 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:03.273165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.18.90.1:43561)
0806 16:12:03.273337 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:03.273342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:03.273466 (+ 124us) server_negotiation.cc:407] Connection header received
0806 16:12:03.273651 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:03.273654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:03.273698 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:03.273783 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:03.274562 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.275090 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.275986 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:03.276216 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:03.276373 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:03.278937 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:03.278940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:03.278942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:03.278970 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:03.280346 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:03.280759 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:03.280761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:03.280762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:03.280837 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:03.281069 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:03.281071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:03.281072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:03.281237 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:03.281336 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:03.281533 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 16:12:03.281640 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:12:03.282958 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 18909
I20260806 16:12:03.290373 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19045
I20260806 16:12:03.296690 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19182
I20260806 16:12:03.302881 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 18812
2026-08-06T16:12:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2297 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[19327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[19327](info): set up 2 sockets
Aug 06 16:12:03 dist-test-slave-0b58 krb5kdc[19327](info): commencing operation
krb5kdc: starting...
W20260806 16:12:04.340962 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:04 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:04.467272 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45051
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:04.583853 19340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:04.584142 19340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:04.584183 19340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:04.588747 19340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:04.588810 19340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:04.588830 19340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:04.588847 19340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:04.588898 19340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:04.594297 19340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:45051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:04.595466 19340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:04.596637 19340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:04.605507 19346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:04.605526 19348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:04.605515 19345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:04.605652 19340 server_base.cc:1034] running on GCE node
I20260806 16:12:04.606304 19340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:04.607420 19340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:04.608582 19340 hybrid_clock.cc:648] HybridClock initialized: now 1786032724608564 us; error 30 us; skew 500 ppm
Aug 06 16:12:04 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032724, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:04.611351 19340 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:04.612357 19340 webserver.cc:460] Webserver started at http://127.18.90.62:32805/ using document root <none> and password file <none>
I20260806 16:12:04.612932 19340 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:04.613022 19340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:04.613263 19340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:04.615391 19340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "231054d2c09d41cea4bb066c917ac121"
format_stamp: "Formatted at 2026-08-06 16:12:04 on dist-test-slave-0b58"
server_key: "894bcd5a5f80d415ec286de61743cbc1"
server_key_iv: "3c45d438e95213350e654508dd18d5bc"
server_key_version: "encryptionkey@0"
I20260806 16:12:04.616001 19340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "231054d2c09d41cea4bb066c917ac121"
format_stamp: "Formatted at 2026-08-06 16:12:04 on dist-test-slave-0b58"
server_key: "894bcd5a5f80d415ec286de61743cbc1"
server_key_iv: "3c45d438e95213350e654508dd18d5bc"
server_key_version: "encryptionkey@0"
I20260806 16:12:04.619963 19340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:12:04.622519 19355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:04.623777 19340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:04.623917 19340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "231054d2c09d41cea4bb066c917ac121"
format_stamp: "Formatted at 2026-08-06 16:12:04 on dist-test-slave-0b58"
server_key: "894bcd5a5f80d415ec286de61743cbc1"
server_key_iv: "3c45d438e95213350e654508dd18d5bc"
server_key_version: "encryptionkey@0"
I20260806 16:12:04.624032 19340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:04.637364 19340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:04.640394 19340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:04.640596 19340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:04.647572 19340 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:45051
I20260806 16:12:04.647585 19407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:45051 every 8 connection(s)
I20260806 16:12:04.648631 19340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:04.651412 19408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:04.654183 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19340
I20260806 16:12:04.654302 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:04.654588 18792 external_mini_cluster.cc:1428] Setting key a361e77075aafe3fc60247cc3d69e1eb
I20260806 16:12:04.657717 19408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121: Bootstrap starting.
I20260806 16:12:04.660451 19408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:04.661304 19408 log.cc:824] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:04.663375 19408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121: No bootstrap required, opened a new log
Aug 06 16:12:04 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:04.666841 19408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231054d2c09d41cea4bb066c917ac121" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45051 } }
I20260806 16:12:04.667063 19408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:04.667124 19408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231054d2c09d41cea4bb066c917ac121, State: Initialized, Role: FOLLOWER
I20260806 16:12:04.667550 19408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231054d2c09d41cea4bb066c917ac121" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45051 } }
I20260806 16:12:04.667697 19408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:04.667790 19408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:04.667910 19408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:04.668865 19408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231054d2c09d41cea4bb066c917ac121" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45051 } }
I20260806 16:12:04.669265 19408 leader_election.cc:302] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231054d2c09d41cea4bb066c917ac121; no voters:
I20260806 16:12:04.669570 19408 leader_election.cc:288] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:04.669700 19413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:04.669931 19413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [term 1 LEADER]: Becoming Leader. State: Replica: 231054d2c09d41cea4bb066c917ac121, State: Running, Role: LEADER
I20260806 16:12:04.670135 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:04.655986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.18.90.62:45051)
0806 16:12:04.656427 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:04.656450 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:04.656487 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:12:04.657124 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:04.657142 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:04.657497 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:04.657783 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:04.658688 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:04.659781 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:04.660986 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:04.661542 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:04.661726 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:04.664445 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:04.664450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:04.664463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:04.664490 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:04.666791 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:04.667344 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:04.667352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:04.667359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:04.667500 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:04.667835 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:04.667838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:04.667840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:04.668352 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:04.668572 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:04.668830 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 16:12:04.669132 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":150,"threads_started":1}
I20260806 16:12:04.670336 19413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231054d2c09d41cea4bb066c917ac121" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45051 } }
I20260806 16:12:04.670823 19408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:04.671496 19415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231054d2c09d41cea4bb066c917ac121. Latest consensus state: current_term: 1 leader_uuid: "231054d2c09d41cea4bb066c917ac121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231054d2c09d41cea4bb066c917ac121" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45051 } } }
I20260806 16:12:04.671734 19415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:04.671994 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231054d2c09d41cea4bb066c917ac121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231054d2c09d41cea4bb066c917ac121" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45051 } } }
I20260806 16:12:04.672096 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:04.672503 19422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:04.675470 19422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:04.680907 19422 catalog_manager.cc:1347] Generated new cluster ID: ec74e8baf7bb4322b93ee498eba83ebb
I20260806 16:12:04.680986 19422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:04.688078 19422 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:04.689358 19422 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:04.697446 19422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231054d2c09d41cea4bb066c917ac121: Generated new TSK 0
I20260806 16:12:04.698320 19422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:04.728374 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45051
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:04.843652 19434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:04.843977 19434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:04.844056 19434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:04.848191 19434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:04.848311 19434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:04.848418 19434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:04.853921 19434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45051
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:04.855132 19434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:04.856225 19434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:04.866478 19440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:04.866526 19434 server_base.cc:1034] running on GCE node
W20260806 16:12:04.866487 19439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:04.866478 19442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:04.867154 19434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:04.867789 19434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:04.868981 19434 hybrid_clock.cc:648] HybridClock initialized: now 1786032724868962 us; error 30 us; skew 500 ppm
Aug 06 16:12:04 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032724, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:04.872026 19434 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:04.873045 19434 webserver.cc:460] Webserver started at http://127.18.90.1:34721/ using document root <none> and password file <none>
I20260806 16:12:04.873647 19434 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:04.873739 19434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:04.873961 19434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:04.876099 19434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "ef79a088cf7149178db1f7cfd7a7f8e0"
format_stamp: "Formatted at 2026-08-06 16:12:04 on dist-test-slave-0b58"
server_key: "1e00b8d6eb0a3a6581c9b33b95a06f63"
server_key_iv: "195d030339732d4c2959565fa8d647e0"
server_key_version: "encryptionkey@0"
I20260806 16:12:04.876754 19434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "ef79a088cf7149178db1f7cfd7a7f8e0"
format_stamp: "Formatted at 2026-08-06 16:12:04 on dist-test-slave-0b58"
server_key: "1e00b8d6eb0a3a6581c9b33b95a06f63"
server_key_iv: "195d030339732d4c2959565fa8d647e0"
server_key_version: "encryptionkey@0"
I20260806 16:12:04.880913 19434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:04.883368 19450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:04.884701 19434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:04.884837 19434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "ef79a088cf7149178db1f7cfd7a7f8e0"
format_stamp: "Formatted at 2026-08-06 16:12:04 on dist-test-slave-0b58"
server_key: "1e00b8d6eb0a3a6581c9b33b95a06f63"
server_key_iv: "195d030339732d4c2959565fa8d647e0"
server_key_version: "encryptionkey@0"
I20260806 16:12:04.885039 19434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:04.898106 19434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:04.901275 19434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:04.901497 19434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:04.902127 19434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:04.902987 19434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:04.903034 19434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:04.903102 19434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:04.903134 19434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:04.912307 19434 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:43577
I20260806 16:12:04.912352 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:43577 every 8 connection(s)
I20260806 16:12:04.913275 19434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:04.916456 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19434
I20260806 16:12:04.916585 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:04.916968 18792 external_mini_cluster.cc:1428] Setting key 342a92fcc120104fabe39911bf8a4549
Aug 06 16:12:04 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032724, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:04.929538 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:04.915105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:60377 (local address 127.18.90.62:45051)
0806 16:12:04.915260 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:04.915263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:04.916130 (+ 867us) server_negotiation.cc:407] Connection header received
0806 16:12:04.918083 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:04.918087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:04.918143 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:04.918250 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:04.919510 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:04.920050 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:04.921884 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:04.922122 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:04.922239 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:04.925305 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:04.925308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:04.925310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:04.925337 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:04.926850 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:04.927574 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:04.927577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:04.927578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:04.927658 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:04.928106 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:04.928109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:04.928110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:04.928601 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:04.928698 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:04.929287 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 16:12:04.929407 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:12:04.930632 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:04.915514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45051 (local address 127.18.90.1:60377)
0806 16:12:04.915966 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:04.916014 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:04.917857 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:04.918446 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:04.918454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:04.918869 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:04.919364 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:04.919378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:04.920181 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:04.920185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:04.921755 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:04.921766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:04.922283 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:04.922287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:04.922476 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:04.923140 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:04.923157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:04.925115 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:04.926980 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:04.926987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:04.927005 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:04.927443 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:04.927754 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:04.927761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:04.927766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:04.927977 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:04.928768 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:04.928775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:04.929048 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 16:12:04.929223 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 16:12:04.929509 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260806 16:12:04.931954 19566 heartbeater.cc:347] Connected to a master server at 127.18.90.62:45051
I20260806 16:12:04.932235 19566 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:04.932796 19566 heartbeater.cc:511] Master 127.18.90.62:45051 requested a full tablet report, sending...
I20260806 16:12:04.934391 19372 ts_manager.cc:194] Registered new tserver with Master: ef79a088cf7149178db1f7cfd7a7f8e0 (127.18.90.1:43577)
I20260806 16:12:04.935550 19372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:60377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:04.947325 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45051
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:05.063731 19571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:05.064004 19571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:05.064050 19571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:05.068095 19571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:05.068166 19571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:05.068231 19571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:05.074131 19571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45051
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:05.075210 19571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:05.076224 19571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:05.086062 19578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:05.086073 19577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:05.086086 19580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:05.086129 19571 server_base.cc:1034] running on GCE node
I20260806 16:12:05.086786 19571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:05.087441 19571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:05.088616 19571 hybrid_clock.cc:648] HybridClock initialized: now 1786032725088577 us; error 51 us; skew 500 ppm
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:05.091565 19571 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:05.092468 19571 webserver.cc:460] Webserver started at http://127.18.90.2:32833/ using document root <none> and password file <none>
I20260806 16:12:05.093009 19571 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:05.093068 19571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:05.093246 19571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:05.095305 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "2c738f8da665457d80fa922eb4e7353b"
format_stamp: "Formatted at 2026-08-06 16:12:05 on dist-test-slave-0b58"
server_key: "f7fa47529ad17f9becbad450af4e4fff"
server_key_iv: "5fd06c71b91eace52f9f05a4ac9fdb91"
server_key_version: "encryptionkey@0"
I20260806 16:12:05.095832 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "2c738f8da665457d80fa922eb4e7353b"
format_stamp: "Formatted at 2026-08-06 16:12:05 on dist-test-slave-0b58"
server_key: "f7fa47529ad17f9becbad450af4e4fff"
server_key_iv: "5fd06c71b91eace52f9f05a4ac9fdb91"
server_key_version: "encryptionkey@0"
I20260806 16:12:05.099620 19571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:05.102097 19588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:05.103351 19571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:12:05.103475 19571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "2c738f8da665457d80fa922eb4e7353b"
format_stamp: "Formatted at 2026-08-06 16:12:05 on dist-test-slave-0b58"
server_key: "f7fa47529ad17f9becbad450af4e4fff"
server_key_iv: "5fd06c71b91eace52f9f05a4ac9fdb91"
server_key_version: "encryptionkey@0"
I20260806 16:12:05.103585 19571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:05.127099 19571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:05.130697 19571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:05.130913 19571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:05.131561 19571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:05.132680 19571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:05.132735 19571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:05.132813 19571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:05.132856 19571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:05.141688 19571 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38397
I20260806 16:12:05.141767 19701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38397 every 8 connection(s)
I20260806 16:12:05.142665 19571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:05.144717 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19571
I20260806 16:12:05.144833 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:05.145131 18792 external_mini_cluster.cc:1428] Setting key ddd06d78b0fb55b1c690fe7a856465d5
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:12:05.157532 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.144513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:60273 (local address 127.18.90.62:45051)
0806 16:12:05.144653 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.144656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.145755 (+ 1099us) server_negotiation.cc:407] Connection header received
0806 16:12:05.147377 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.147380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.147439 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.147514 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:05.148760 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.149303 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.150464 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.150703 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.150814 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.153699 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:05.153703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:05.153705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:05.153749 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:05.155357 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.155848 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.155851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.155852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.155945 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.156348 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.156352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.156354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.156556 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:05.156665 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.157200 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.157348 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:05.158561 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.145065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45051 (local address 127.18.90.2:60273)
0806 16:12:05.145601 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.145660 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.147151 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.147693 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.147699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.148060 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:05.148613 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.148627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.149433 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.149437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.150337 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.150347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.150864 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.150868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.151080 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.151746 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:05.151764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:05.153506 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:05.155478 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:05.155483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:05.155496 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:05.155751 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:05.156056 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:05.156059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:05.156062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:05.156215 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:05.156664 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:05.156672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:05.156938 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.157129 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.157481 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":144,"threads_started":1}
I20260806 16:12:05.159794 19702 heartbeater.cc:347] Connected to a master server at 127.18.90.62:45051
I20260806 16:12:05.160051 19702 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:05.160665 19702 heartbeater.cc:511] Master 127.18.90.62:45051 requested a full tablet report, sending...
I20260806 16:12:05.161801 19372 ts_manager.cc:194] Registered new tserver with Master: 2c738f8da665457d80fa922eb4e7353b (127.18.90.2:38397)
I20260806 16:12:05.162707 19372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:60273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:05.175170 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45051
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:05.292510 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:05.292793 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:05.292840 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:05.296921 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:05.296989 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:05.297052 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:05.302452 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45051
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:05.303529 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:05.304593 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:05.314569 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:05.314587 19713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:05.314734 19708 server_base.cc:1034] running on GCE node
W20260806 16:12:05.314913 19714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:05.315353 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:05.315976 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:05.317134 19708 hybrid_clock.cc:648] HybridClock initialized: now 1786032725317087 us; error 61 us; skew 500 ppm
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:05.319933 19708 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:05.320940 19708 webserver.cc:460] Webserver started at http://127.18.90.3:36079/ using document root <none> and password file <none>
I20260806 16:12:05.321552 19708 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:05.321647 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:05.321883 19708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:05.324024 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "2c570a0bb69745b88e4f23636e413bb4"
format_stamp: "Formatted at 2026-08-06 16:12:05 on dist-test-slave-0b58"
server_key: "d27b45cd6ee5df3611df579d0e5d34a1"
server_key_iv: "a6c7192bcc82dd911f8c3cba58964595"
server_key_version: "encryptionkey@0"
I20260806 16:12:05.324699 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "2c570a0bb69745b88e4f23636e413bb4"
format_stamp: "Formatted at 2026-08-06 16:12:05 on dist-test-slave-0b58"
server_key: "d27b45cd6ee5df3611df579d0e5d34a1"
server_key_iv: "a6c7192bcc82dd911f8c3cba58964595"
server_key_version: "encryptionkey@0"
I20260806 16:12:05.328711 19708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:05.331262 19724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:05.332758 19708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:05.332894 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "2c570a0bb69745b88e4f23636e413bb4"
format_stamp: "Formatted at 2026-08-06 16:12:05 on dist-test-slave-0b58"
server_key: "d27b45cd6ee5df3611df579d0e5d34a1"
server_key_iv: "a6c7192bcc82dd911f8c3cba58964595"
server_key_version: "encryptionkey@0"
I20260806 16:12:05.333016 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:05.353538 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:05.356595 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:05.356810 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:05.357431 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:05.358286 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:05.358361 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:05.358429 19708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:05.358477 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:05.367599 19708 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:37143
I20260806 16:12:05.367624 19837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:37143 every 8 connection(s)
I20260806 16:12:05.368605 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:05.372411 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19708
I20260806 16:12:05.372527 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:05.372807 18792 external_mini_cluster.cc:1428] Setting key f8516fe744cff51c3bf57db724771e8b
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:05.382385 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.370408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:52697 (local address 127.18.90.62:45051)
0806 16:12:05.370553 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.370557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.371424 (+ 867us) server_negotiation.cc:407] Connection header received
0806 16:12:05.373043 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.373047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.373110 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.373199 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:05.374578 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.375192 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.376395 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.376610 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.376687 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.379006 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:05.379009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:05.379011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:05.379036 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:05.380520 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.380961 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.380964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.380965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.381038 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.381353 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.381356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.381357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.381552 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:05.381642 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.382141 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.382256 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:12:05.383351 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.370813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45051 (local address 127.18.90.3:52697)
0806 16:12:05.371264 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.371320 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.372796 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.373393 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.373414 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.373825 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:05.374414 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.374434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.375345 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.375348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.376233 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.376243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.376736 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.376739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.376883 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.377420 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:05.377436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:05.378851 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:05.380623 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:05.380628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:05.380639 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:05.380881 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:05.381112 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:05.381114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:05.381115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:05.381242 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:05.381669 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:05.381674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:05.381934 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.382095 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.382356 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20260806 16:12:05.384603 19838 heartbeater.cc:347] Connected to a master server at 127.18.90.62:45051
I20260806 16:12:05.384873 19838 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:05.385376 19838 heartbeater.cc:511] Master 127.18.90.62:45051 requested a full tablet report, sending...
I20260806 16:12:05.386492 19372 ts_manager.cc:194] Registered new tserver with Master: 2c570a0bb69745b88e4f23636e413bb4 (127.18.90.3:37143)
I20260806 16:12:05.387069 19372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:52697
I20260806 16:12:05.398891 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:12:05.411314 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.403141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.18.90.62:45051)
0806 16:12:05.403291 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.403297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.403471 (+ 174us) server_negotiation.cc:407] Connection header received
0806 16:12:05.403650 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.403654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.403697 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.403791 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:05.404596 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.405125 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.406243 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.406477 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.406591 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.407989 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:05.407999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:05.408002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:05.408026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:05.409520 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.409984 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.409988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.409989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.410076 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.410367 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.410370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.410372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.410607 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:05.410749 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.411015 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.411142 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 16:12:05.417168 19372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:05.420490 19372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:05.436688 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.430253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.18.90.2:38397)
0806 16:12:05.431128 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.431145 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.431173 (+ 28us) server_negotiation.cc:407] Connection header received
0806 16:12:05.431370 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.431376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.431597 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.431757 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:05.432082 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.433301 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.435239 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.435929 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.436056 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.436211 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.436377 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.436471 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":120,"threads_started":1}
I20260806 16:12:05.436684 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.430093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38397 (local address 127.0.0.1:44708)
0806 16:12:05.430962 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.431017 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.431152 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.431744 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.431750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.431773 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:05.431944 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.431952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.433443 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.433447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.435093 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.435104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.436067 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.436071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.436176 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.436212 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.436346 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.436470 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":169,"thread_start_us":98,"threads_started":1}
I20260806 16:12:05.436724 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.430407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43577 (local address 127.0.0.1:41806)
0806 16:12:05.431429 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.431445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.431537 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.432274 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.432278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.432298 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:05.432392 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.432400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.433506 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.433510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.435058 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.435068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.436281 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.436285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.436382 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.436404 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.436462 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.436538 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":82,"thread_start_us":670,"threads_started":1}
I20260806 16:12:05.436932 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.430103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:37143 (local address 127.0.0.1:45776)
0806 16:12:05.431472 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.431487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.431572 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.432585 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.432589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.432608 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:05.432695 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.432701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.434100 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.434104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.435255 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.435263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.436524 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.436527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.436654 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.436675 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.436742 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.436820 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":25,"spinlock_wait_cycles":9088,"thread_start_us":97,"threads_started":1}
I20260806 16:12:05.437301 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.430837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.18.90.1:43577)
0806 16:12:05.431798 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.431814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.431839 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:12:05.431914 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.431921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.432106 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.432301 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:05.432491 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.433339 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.435186 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.436102 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.436834 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.436991 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.437051 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.437158 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":116,"threads_started":1}
I20260806 16:12:05.437510 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.430963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.18.90.3:37143)
0806 16:12:05.432064 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.432077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.432099 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:12:05.432172 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.432177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.432400 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.432582 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:05.432820 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.433935 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.435373 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.436148 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.436283 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.436430 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.437226 (+ 796us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.437326 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":202,"threads_started":1}
I20260806 16:12:05.440362 19772 tablet_service.cc:1467] Processing CreateTablet for tablet 97d0fc1ad48d4103a789d63444394474 (DEFAULT_TABLE table=test-table [id=133538accd384d6c95dcd6b406b2b3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:05.440320 19636 tablet_service.cc:1467] Processing CreateTablet for tablet 97d0fc1ad48d4103a789d63444394474 (DEFAULT_TABLE table=test-table [id=133538accd384d6c95dcd6b406b2b3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:05.441367 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d0fc1ad48d4103a789d63444394474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:05.441390 19636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d0fc1ad48d4103a789d63444394474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:05.442330 19500 tablet_service.cc:1467] Processing CreateTablet for tablet 97d0fc1ad48d4103a789d63444394474 (DEFAULT_TABLE table=test-table [id=133538accd384d6c95dcd6b406b2b3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:05.443329 19500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d0fc1ad48d4103a789d63444394474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:05.447575 19857 tablet_bootstrap.cc:492] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4: Bootstrap starting.
I20260806 16:12:05.449574 19858 tablet_bootstrap.cc:492] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b: Bootstrap starting.
I20260806 16:12:05.449846 19857 tablet_bootstrap.cc:654] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:05.450904 19859 tablet_bootstrap.cc:492] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0: Bootstrap starting.
I20260806 16:12:05.451136 19857 log.cc:824] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:05.452368 19858 tablet_bootstrap.cc:654] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:05.453218 19858 log.cc:824] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:05.453646 19857 tablet_bootstrap.cc:492] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4: No bootstrap required, opened a new log
I20260806 16:12:05.453673 19859 tablet_bootstrap.cc:654] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:05.453850 19857 ts_tablet_manager.cc:1397] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260806 16:12:05.454553 19859 log.cc:824] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:05.455382 19858 tablet_bootstrap.cc:492] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b: No bootstrap required, opened a new log
I20260806 16:12:05.455579 19858 ts_tablet_manager.cc:1397] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:12:05.456645 19859 tablet_bootstrap.cc:492] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0: No bootstrap required, opened a new log
I20260806 16:12:05.456827 19859 ts_tablet_manager.cc:1397] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 16:12:05.457404 19857 raft_consensus.cc:348] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.457679 19857 raft_consensus.cc:374] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:05.457754 19857 raft_consensus.cc:729] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c570a0bb69745b88e4f23636e413bb4, State: Initialized, Role: FOLLOWER
I20260806 16:12:05.458248 19857 consensus_queue.cc:260] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.458930 19838 heartbeater.cc:503] Master 127.18.90.62:45051 was elected leader, sending a full tablet report...
I20260806 16:12:05.459313 19857 ts_tablet_manager.cc:1428] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260806 16:12:05.460760 19858 raft_consensus.cc:348] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.461051 19858 raft_consensus.cc:374] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:05.461156 19858 raft_consensus.cc:729] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c738f8da665457d80fa922eb4e7353b, State: Initialized, Role: FOLLOWER
I20260806 16:12:05.461445 19859 raft_consensus.cc:348] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.461699 19859 raft_consensus.cc:374] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:05.461621 19858 consensus_queue.cc:260] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.461805 19859 raft_consensus.cc:729] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef79a088cf7149178db1f7cfd7a7f8e0, State: Initialized, Role: FOLLOWER
I20260806 16:12:05.462297 19859 consensus_queue.cc:260] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.462388 19702 heartbeater.cc:503] Master 127.18.90.62:45051 was elected leader, sending a full tablet report...
I20260806 16:12:05.462707 19858 ts_tablet_manager.cc:1428] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 16:12:05.463318 19566 heartbeater.cc:503] Master 127.18.90.62:45051 was elected leader, sending a full tablet report...
I20260806 16:12:05.463579 19863 raft_consensus.cc:482] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:05.463739 19863 raft_consensus.cc:504] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.464515 19859 ts_tablet_manager.cc:1428] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 16:12:05.464946 19863 leader_election.cc:288] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c738f8da665457d80fa922eb4e7353b (127.18.90.2:38397), ef79a088cf7149178db1f7cfd7a7f8e0 (127.18.90.1:43577)
I20260806 16:12:05.469589 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.465656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:41465 (local address 127.18.90.1:43577)
0806 16:12:05.465772 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.465777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.465791 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:05.465835 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.465838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.465900 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.465981 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:05.466339 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.467038 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.468449 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.469065 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.469152 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.469202 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.469438 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.469491 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 16:12:05.469694 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.465157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43577 (local address 127.18.90.3:41465)
0806 16:12:05.465276 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.465294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.465384 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.466012 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.466015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.466035 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:05.466196 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.466207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.467173 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.467176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.468322 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.468339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.469196 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.469199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.469308 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.469335 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.469425 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.469530 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 16:12:05.470110 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.465335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:46873 (local address 127.18.90.2:38397)
0806 16:12:05.465572 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.465577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.465665 (+ 88us) server_negotiation.cc:407] Connection header received
0806 16:12:05.466244 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.466248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.466295 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.466341 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:05.466745 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.467467 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.468928 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.469549 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.469648 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.469703 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.469950 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.470025 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260806 16:12:05.470283 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.465171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38397 (local address 127.18.90.3:46873)
0806 16:12:05.465562 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:05.465578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:05.465687 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:05.466472 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:05.466474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:05.466491 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:05.466609 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.466616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.467605 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.467609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.468769 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:05.468780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.469680 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.469685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:05.469817 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.469841 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:12:05.469917 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 16:12:05.469988 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":94,"thread_start_us":104,"threads_started":1}
I20260806 16:12:05.470436 19520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0fc1ad48d4103a789d63444394474" candidate_uuid: "2c570a0bb69745b88e4f23636e413bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" is_pre_election: true
I20260806 16:12:05.470775 19520 raft_consensus.cc:2454] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c570a0bb69745b88e4f23636e413bb4 in term 0.
I20260806 16:12:05.471001 19656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0fc1ad48d4103a789d63444394474" candidate_uuid: "2c570a0bb69745b88e4f23636e413bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c738f8da665457d80fa922eb4e7353b" is_pre_election: true
I20260806 16:12:05.471277 19656 raft_consensus.cc:2454] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c570a0bb69745b88e4f23636e413bb4 in term 0.
I20260806 16:12:05.471321 19727 leader_election.cc:302] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c570a0bb69745b88e4f23636e413bb4, ef79a088cf7149178db1f7cfd7a7f8e0; no voters:
I20260806 16:12:05.471647 19863 raft_consensus.cc:2781] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:05.471742 19863 raft_consensus.cc:482] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:05.471807 19863 raft_consensus.cc:3037] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:05.473038 19863 raft_consensus.cc:504] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.473387 19863 leader_election.cc:288] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [CANDIDATE]: Term 1 election: Requested vote from peers 2c738f8da665457d80fa922eb4e7353b (127.18.90.2:38397), ef79a088cf7149178db1f7cfd7a7f8e0 (127.18.90.1:43577)
I20260806 16:12:05.473781 19520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0fc1ad48d4103a789d63444394474" candidate_uuid: "2c570a0bb69745b88e4f23636e413bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0"
I20260806 16:12:05.473799 19656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0fc1ad48d4103a789d63444394474" candidate_uuid: "2c570a0bb69745b88e4f23636e413bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c738f8da665457d80fa922eb4e7353b"
I20260806 16:12:05.473910 19656 raft_consensus.cc:3037] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:05.473923 19520 raft_consensus.cc:3037] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:05.475322 19656 raft_consensus.cc:2454] T 97d0fc1ad48d4103a789d63444394474 P 2c738f8da665457d80fa922eb4e7353b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c570a0bb69745b88e4f23636e413bb4 in term 1.
I20260806 16:12:05.475395 19520 raft_consensus.cc:2454] T 97d0fc1ad48d4103a789d63444394474 P ef79a088cf7149178db1f7cfd7a7f8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c570a0bb69745b88e4f23636e413bb4 in term 1.
I20260806 16:12:05.475704 19727 leader_election.cc:302] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c570a0bb69745b88e4f23636e413bb4, ef79a088cf7149178db1f7cfd7a7f8e0; no voters:
I20260806 16:12:05.475917 19863 raft_consensus.cc:2781] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:05.476112 19863 raft_consensus.cc:686] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 2c570a0bb69745b88e4f23636e413bb4, State: Running, Role: LEADER
I20260806 16:12:05.476500 19863 consensus_queue.cc:237] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } }
I20260806 16:12:05.479630 19372 catalog_manager.cc:5495] T 97d0fc1ad48d4103a789d63444394474 P 2c570a0bb69745b88e4f23636e413bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c570a0bb69745b88e4f23636e413bb4 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "2c570a0bb69745b88e4f23636e413bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c738f8da665457d80fa922eb4e7353b" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c570a0bb69745b88e4f23636e413bb4" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef79a088cf7149178db1f7cfd7a7f8e0" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43577 } health_report { overall_health: UNKNOWN } } }
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:05.496905 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.487081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.18.90.1:43577)
0806 16:12:05.487232 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.487237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.487407 (+ 170us) server_negotiation.cc:407] Connection header received
0806 16:12:05.487531 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.487533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.487574 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.487640 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:05.488414 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.489093 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.490130 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.490358 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.490448 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.493302 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:05.493314 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:05.493329 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:05.493354 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:05.495047 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.495469 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.495474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.495476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.495564 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.495799 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.495801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.495802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.496058 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:05.496218 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.496493 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.496712 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:05.525467 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.516804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.18.90.1:43577)
0806 16:12:05.517002 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.517006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.517144 (+ 138us) server_negotiation.cc:407] Connection header received
0806 16:12:05.517207 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.517209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.517256 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.517349 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:05.518066 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.518628 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.519492 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.519737 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.519823 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.522300 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:05.522303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:05.522305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:05.522329 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:05.523808 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.524229 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.524231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.524232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.524368 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.524611 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.524613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.524614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.524777 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:05.524875 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.525128 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.525230 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:05.553879 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:05.545216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.18.90.1:43577)
0806 16:12:05.545396 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:05.545401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:05.545534 (+ 133us) server_negotiation.cc:407] Connection header received
0806 16:12:05.545710 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:05.545715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:05.545765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:05.545858 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:05.546607 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.547122 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.548011 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:05.548237 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:05.548404 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:05.550886 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:05.550889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:05.550891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:05.550915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:05.552103 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.552620 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.552624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.552626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.552711 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:05.552999 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:05.553002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:05.553003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:05.553188 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:05.553326 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:05.553582 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 16:12:05.553703 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 16:12:05.555303 19478 tablet_service.cc:2908] Rejecting scan request for tablet 97d0fc1ad48d4103a789d63444394474: Uninitialized: clean time has not yet been initialized
I20260806 16:12:05.557523 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19434
I20260806 16:12:05.564865 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19571
I20260806 16:12:05.572109 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19708
I20260806 16:12:05.578743 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19340
2026-08-06T16:12:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2276 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19881](info): set up 2 sockets
Aug 06 16:12:05 dist-test-slave-0b58 krb5kdc[19881](info): commencing operation
krb5kdc: starting...
W20260806 16:12:06.637027 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:06 dist-test-slave-0b58 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:06.771150 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:34731
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:34731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:06.888180 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:06.888558 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:06.888650 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:06.893055 19894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:06.893193 19894 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 16:12:06.893234 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:06.893312 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:06.893354 19894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:06.893392 19894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:06.899847 19894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:34731
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:34731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:06.901106 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:06.902354 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:06.911542 19900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:06.911520 19902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:06.911536 19899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:06.911751 19894 server_base.cc:1034] running on GCE node
I20260806 16:12:06.912520 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:06.913728 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:06.914929 19894 hybrid_clock.cc:648] HybridClock initialized: now 1786032726914916 us; error 30 us; skew 500 ppm
Aug 06 16:12:06 dist-test-slave-0b58 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032726, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:06.917810 19894 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:06.918792 19894 webserver.cc:460] Webserver started at http://127.18.90.62:37267/ using document root <none> and password file <none>
I20260806 16:12:06.919405 19894 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:06.919502 19894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:06.919741 19894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:06.921967 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "6eaded62a40d40088a5ebc5a7f28e546"
format_stamp: "Formatted at 2026-08-06 16:12:06 on dist-test-slave-0b58"
server_key: "7151fe7aab66d2615f050bade5b63fd1"
server_key_iv: "1183bacd1edf30a8b8d7f29e61b62f6b"
server_key_version: "encryptionkey@0"
I20260806 16:12:06.922580 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "6eaded62a40d40088a5ebc5a7f28e546"
format_stamp: "Formatted at 2026-08-06 16:12:06 on dist-test-slave-0b58"
server_key: "7151fe7aab66d2615f050bade5b63fd1"
server_key_iv: "1183bacd1edf30a8b8d7f29e61b62f6b"
server_key_version: "encryptionkey@0"
I20260806 16:12:06.926693 19894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:06.929351 19909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:06.930673 19894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:06.930827 19894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "6eaded62a40d40088a5ebc5a7f28e546"
format_stamp: "Formatted at 2026-08-06 16:12:06 on dist-test-slave-0b58"
server_key: "7151fe7aab66d2615f050bade5b63fd1"
server_key_iv: "1183bacd1edf30a8b8d7f29e61b62f6b"
server_key_version: "encryptionkey@0"
I20260806 16:12:06.930955 19894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:06.948544 19894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:06.951809 19894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:06.952041 19894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:06.960598 19894 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:34731
I20260806 16:12:06.960618 19971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:34731 every 8 connection(s)
I20260806 16:12:06.961642 19894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:06.964586 19972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:06.968698 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19894
I20260806 16:12:06.968842 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:06.969091 18792 external_mini_cluster.cc:1428] Setting key 5b7bd450814cf84b752f2187cf9c15fb
I20260806 16:12:06.970783 19972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: Bootstrap starting.
I20260806 16:12:06.973183 19972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:06.973960 19972 log.cc:824] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:06.975888 19972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: No bootstrap required, opened a new log
Aug 06 16:12:06 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:06.979918 19972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eaded62a40d40088a5ebc5a7f28e546" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34731 } }
I20260806 16:12:06.980175 19972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:06.980257 19972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eaded62a40d40088a5ebc5a7f28e546, State: Initialized, Role: FOLLOWER
I20260806 16:12:06.980733 19972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eaded62a40d40088a5ebc5a7f28e546" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34731 } }
I20260806 16:12:06.980885 19972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:06.980973 19972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:06.981094 19972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:06.982003 19972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eaded62a40d40088a5ebc5a7f28e546" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34731 } }
I20260806 16:12:06.982365 19972 leader_election.cc:302] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eaded62a40d40088a5ebc5a7f28e546; no voters:
I20260806 16:12:06.982681 19972 leader_election.cc:288] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:06.982813 19977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:06.983096 19977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [term 1 LEADER]: Becoming Leader. State: Replica: 6eaded62a40d40088a5ebc5a7f28e546, State: Running, Role: LEADER
I20260806 16:12:06.983462 19977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eaded62a40d40088a5ebc5a7f28e546" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34731 } }
I20260806 16:12:06.984210 19972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:06.984745 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:06.970612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.18.90.62:34731)
0806 16:12:06.971128 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:06.971144 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:06.971203 (+ 59us) server_negotiation.cc:407] Connection header received
0806 16:12:06.971884 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:06.971902 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:06.972248 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:06.972627 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:06.973536 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:06.974602 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:06.975492 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:06.975966 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:06.976145 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:06.978994 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:06.978999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:06.979014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:06.979040 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:06.981132 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:06.981673 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:06.981679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:06.981685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:06.981821 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:06.982118 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:06.982121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:06.982123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:06.982602 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:06.982884 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:06.983262 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 16:12:06.983576 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":157,"threads_started":1}
I20260806 16:12:06.985442 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eaded62a40d40088a5ebc5a7f28e546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eaded62a40d40088a5ebc5a7f28e546" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34731 } } }
I20260806 16:12:06.985577 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:06.986021 19986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:06.984733 19977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eaded62a40d40088a5ebc5a7f28e546. Latest consensus state: current_term: 1 leader_uuid: "6eaded62a40d40088a5ebc5a7f28e546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eaded62a40d40088a5ebc5a7f28e546" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34731 } } }
I20260806 16:12:06.986299 19977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:06.989557 19986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:06.996019 19986 catalog_manager.cc:1347] Generated new cluster ID: edd1c8a42f09427397dba89fec5dcf99
I20260806 16:12:06.996110 19986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:07.028126 19986 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:07.029302 19986 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:07.039456 19986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: Generated new TSK 0
I20260806 16:12:07.040374 19986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:07.075639 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34731
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:07.194329 19998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:07.194669 19998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:07.194756 19998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:07.198957 19998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:07.199069 19998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:07.199147 19998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:07.199231 19998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:07.205199 19998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:07.206418 19998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:07.207531 19998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:07.217466 20003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:07.217649 19998 server_base.cc:1034] running on GCE node
W20260806 16:12:07.217528 20004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:07.217486 20006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:07.218313 19998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:07.219173 19998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:07.220331 19998 hybrid_clock.cc:648] HybridClock initialized: now 1786032727220317 us; error 38 us; skew 500 ppm
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:07.223272 19998 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:07.224174 19998 webserver.cc:460] Webserver started at http://127.18.90.1:43575/ using document root <none> and password file <none>
I20260806 16:12:07.224821 19998 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:07.224887 19998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:07.225061 19998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:07.227175 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "9bb706bea8114528b58b37d9dc2d6126"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "c150c4a5b32fe29cf2491f47d51f4141"
server_key_iv: "247840eee945649d0b7b930a18006f5c"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.227710 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "9bb706bea8114528b58b37d9dc2d6126"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "c150c4a5b32fe29cf2491f47d51f4141"
server_key_iv: "247840eee945649d0b7b930a18006f5c"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.231596 19998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:07.234263 20013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:07.235615 19998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:12:07.235785 19998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "9bb706bea8114528b58b37d9dc2d6126"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "c150c4a5b32fe29cf2491f47d51f4141"
server_key_iv: "247840eee945649d0b7b930a18006f5c"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.235929 19998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:07.254413 19998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:07.258127 19998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:07.258358 19998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:07.259905 19998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:07.259956 19998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:07.259999 19998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:07.260058 19998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:07.271435 19998 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:43113
I20260806 16:12:07.271507 20130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:43113 every 8 connection(s)
I20260806 16:12:07.272665 19998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:07.273650 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 19998
I20260806 16:12:07.273787 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:07.274132 18792 external_mini_cluster.cc:1428] Setting key eb7aee8f9905c8b6d863356dff356b6b
I20260806 16:12:07.275640 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.261026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:33399 (local address 127.18.90.62:34731)
0806 16:12:07.261162 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.261166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.262310 (+ 1144us) server_negotiation.cc:407] Connection header received
0806 16:12:07.263481 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.263488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.263562 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.263668 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.265176 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.265735 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.267426 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.267678 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.267807 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.271354 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.271358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.271361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.271398 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.273148 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.273856 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.273860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.273862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.273970 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.274359 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.274364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.274366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.274620 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.274737 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.275310 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.275459 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:12:07.276569 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.261643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.18.90.1:33399)
0806 16:12:07.262144 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.262193 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.263234 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.263922 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.263937 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.264418 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:07.265012 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.265026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.265877 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.265880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.267285 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.267296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.267867 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.267871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.268086 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.269433 (+ 1347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:07.269451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:07.271160 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:07.273324 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.273332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.273346 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.273693 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.274063 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.274065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.274066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.274244 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.274736 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:07.274743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:07.275053 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.275260 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.275612 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260806 16:12:07.279824 20131 heartbeater.cc:347] Connected to a master server at 127.18.90.62:34731
I20260806 16:12:07.279984 20131 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:07.280544 20131 heartbeater.cc:511] Master 127.18.90.62:34731 requested a full tablet report, sending...
I20260806 16:12:07.282300 19925 ts_manager.cc:194] Registered new tserver with Master: 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1:43113)
I20260806 16:12:07.283989 19925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:33399
I20260806 16:12:07.288465 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.280079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:34415 (local address 127.18.90.62:34731)
0806 16:12:07.280206 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.280210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.280513 (+ 303us) server_negotiation.cc:407] Connection header received
0806 16:12:07.280646 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.280651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.280707 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.280778 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.281389 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.281939 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.282884 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.283121 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.283254 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.284710 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.284714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.284718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.284746 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.286680 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.287206 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.287209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.287210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.287291 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.287608 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.287610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.287611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.287774 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.287850 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.288163 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.288332 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
I20260806 16:12:07.288549 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.279989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.18.90.1:34415)
0806 16:12:07.280424 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.280440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.280538 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.280820 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.280823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.281047 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:07.281205 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.281216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.282077 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.282081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.282756 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.282766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.283262 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.283265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.283365 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.283948 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:07.283959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:07.284460 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:07.286808 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.286812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.286814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.287096 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.287389 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.287396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.287399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.287513 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.287878 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:07.287882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:07.287988 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.288117 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.288317 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:07.309463 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34731
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:07.426623 20140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:07.426914 20140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:07.426960 20140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:07.431092 20140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:07.431156 20140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:07.431197 20140 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:07.431231 20140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:07.437023 20140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:07.438103 20140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:07.439168 20140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:07.449039 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:07.449048 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:07.449048 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:07.449280 20140 server_base.cc:1034] running on GCE node
I20260806 16:12:07.449765 20140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:07.450419 20140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:07.451609 20140 hybrid_clock.cc:648] HybridClock initialized: now 1786032727451589 us; error 31 us; skew 500 ppm
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:07.458590 20140 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:07.459592 20140 webserver.cc:460] Webserver started at http://127.18.90.2:36069/ using document root <none> and password file <none>
I20260806 16:12:07.460206 20140 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:07.460335 20140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:07.460592 20140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:07.462772 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "8b8699b62042454890045979d5fe22ec"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "0d439b51f89ca1cae33deb6f4c9a8c14"
server_key_iv: "18dca23c39e0a5d039c5f1fb274ae5f2"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.463398 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "8b8699b62042454890045979d5fe22ec"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "0d439b51f89ca1cae33deb6f4c9a8c14"
server_key_iv: "18dca23c39e0a5d039c5f1fb274ae5f2"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.467479 20140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 16:12:07.470146 20155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:07.471481 20140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:07.471628 20140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "8b8699b62042454890045979d5fe22ec"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "0d439b51f89ca1cae33deb6f4c9a8c14"
server_key_iv: "18dca23c39e0a5d039c5f1fb274ae5f2"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.471750 20140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:07.491739 20140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:07.495239 20140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:07.495471 20140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:07.497143 20140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:07.497218 20140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:07.497320 20140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:07.497372 20140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:07.508617 20140 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38187
I20260806 16:12:07.508924 20273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38187 every 8 connection(s)
I20260806 16:12:07.509634 20140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:07.511547 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.498057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:35331 (local address 127.18.90.62:34731)
0806 16:12:07.498194 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.498198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.499274 (+ 1076us) server_negotiation.cc:407] Connection header received
0806 16:12:07.500389 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.500393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.500467 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.500516 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.502054 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.502701 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.503951 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.504202 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.504366 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.507643 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.507647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.507649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.507675 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.509357 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.509976 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.509980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.509982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.510088 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.510467 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.510472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.510474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.510669 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.510776 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.511254 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.511385 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:12:07.512409 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.498614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.18.90.2:35331)
0806 16:12:07.499122 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.499167 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.500077 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.500994 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.501006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.501362 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:07.501910 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.501926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.502835 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.502839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.503821 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.503831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.504430 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.504459 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.504645 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.505313 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:07.505330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:07.507461 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:07.509484 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.509490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.509504 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.509874 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.510202 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.510205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.510207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.510379 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.510778 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:07.510785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:07.511022 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.511212 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.511538 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260806 16:12:07.513635 20274 heartbeater.cc:347] Connected to a master server at 127.18.90.62:34731
I20260806 16:12:07.513772 20274 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:07.514330 20274 heartbeater.cc:511] Master 127.18.90.62:34731 requested a full tablet report, sending...
I20260806 16:12:07.515430 19925 ts_manager.cc:194] Registered new tserver with Master: 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187)
I20260806 16:12:07.516149 19925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:35331
I20260806 16:12:07.517176 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 20140
I20260806 16:12:07.517280 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:07.517566 18792 external_mini_cluster.cc:1428] Setting key 2769b17bd2b68be0c917c14566b0a63e
I20260806 16:12:07.523770 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.515875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.18.90.2:55775)
0806 16:12:07.516183 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.516199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.516400 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.516677 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.516680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.516888 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:07.516987 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.516993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.517886 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.517892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.518711 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.518724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.519231 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.519234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.519310 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.519870 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:07.519888 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:07.520228 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:07.522064 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.522070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.522072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.522375 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.522695 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.522697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.522698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.522778 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.523293 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:07.523295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:07.523374 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.523463 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.523578 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260806 16:12:07.523923 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.515924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:55775 (local address 127.18.90.62:34731)
0806 16:12:07.516061 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.516065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.516328 (+ 263us) server_negotiation.cc:407] Connection header received
0806 16:12:07.516508 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.516512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.516562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.516638 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.517105 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.517761 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.518822 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.519116 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.519222 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.520419 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.520426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.520432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.520464 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.521943 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.522498 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.522504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.522509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.522602 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.522868 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.522874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.522878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.523195 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.523314 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.523549 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.523688 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:07.547780 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34731
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:07.661705 20284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:07.662020 20284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:07.662101 20284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:07.666743 20284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:07.666831 20284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:07.666900 20284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:07.666966 20284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:07.672389 20284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:07.673527 20284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:07.674574 20284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:07.684674 20289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:07.684796 20284 server_base.cc:1034] running on GCE node
W20260806 16:12:07.684648 20292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:07.684965 20290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:07.685427 20284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:07.686196 20284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:07.687378 20284 hybrid_clock.cc:648] HybridClock initialized: now 1786032727687363 us; error 27 us; skew 500 ppm
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:07.690339 20284 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:07.691308 20284 webserver.cc:460] Webserver started at http://127.18.90.3:33565/ using document root <none> and password file <none>
I20260806 16:12:07.691898 20284 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:07.691962 20284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:07.692200 20284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:07.694410 20284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "12e1c3df8f7a4529b68b75caa900ea57"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "f54637256df2818d16b60cf874e982b4"
server_key_iv: "13fb56a6747166ceb335f8b462b29c4b"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.695000 20284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "12e1c3df8f7a4529b68b75caa900ea57"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "f54637256df2818d16b60cf874e982b4"
server_key_iv: "13fb56a6747166ceb335f8b462b29c4b"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.698865 20284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:07.701295 20301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:07.702512 20284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:07.702656 20284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "12e1c3df8f7a4529b68b75caa900ea57"
format_stamp: "Formatted at 2026-08-06 16:12:07 on dist-test-slave-0b58"
server_key: "f54637256df2818d16b60cf874e982b4"
server_key_iv: "13fb56a6747166ceb335f8b462b29c4b"
server_key_version: "encryptionkey@0"
I20260806 16:12:07.702776 20284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:07.718936 20284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:07.722504 20284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:07.722733 20284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:07.724350 20284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:07.724418 20284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:07.724489 20284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:07.724540 20284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:07.735652 20284 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:44853
I20260806 16:12:07.735709 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:44853 every 8 connection(s)
I20260806 16:12:07.736931 20284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:07.739480 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.725111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:38311 (local address 127.18.90.62:34731)
0806 16:12:07.725287 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.725293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.726693 (+ 1400us) server_negotiation.cc:407] Connection header received
0806 16:12:07.727836 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.727839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.727902 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.728009 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.729752 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.730328 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.732249 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.732627 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.732776 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.735822 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.735826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.735829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.735859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.737566 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.738106 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.738109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.738111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.738197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.738457 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.738459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.738460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.738644 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.738730 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.739246 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.739355 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:12:07.740465 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.726038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.18.90.3:38311)
0806 16:12:07.726535 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.726588 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.727601 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.728214 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.728225 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.728824 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:07.729585 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.729619 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.730460 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.730464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.732055 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.732068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.732801 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.732804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.732948 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.733643 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:07.733686 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:07.735613 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:07.737714 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.737721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.737737 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.738005 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.738265 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.738267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.738268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.738382 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.738775 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:07.738782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:07.739027 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.739218 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.739495 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":151,"threads_started":1}
I20260806 16:12:07.741698 20418 heartbeater.cc:347] Connected to a master server at 127.18.90.62:34731
I20260806 16:12:07.741865 20418 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:07.742372 20418 heartbeater.cc:511] Master 127.18.90.62:34731 requested a full tablet report, sending...
I20260806 16:12:07.743450 19926 ts_manager.cc:194] Registered new tserver with Master: 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853)
I20260806 16:12:07.744242 19926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:38311
I20260806 16:12:07.745144 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 20284
I20260806 16:12:07.745268 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:07.745551 18792 external_mini_cluster.cc:1428] Setting key df6c1d0f47d8aba73c9c26d25ec3a89e
I20260806 16:12:07.747541 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:12:07.751569 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.743885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.18.90.3:49881)
0806 16:12:07.744327 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.744343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.744472 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.744710 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.744712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.744931 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:07.745023 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.745028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.746109 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.746112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.746872 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.746885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.747326 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.747330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.747419 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.747963 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:07.747984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:07.748380 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:07.750161 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.750163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.750165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.750360 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.750656 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:07.750659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:07.750660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:07.750739 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:07.751175 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:07.751178 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:07.751248 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.751328 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.751432 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":115,"threads_started":1}
I20260806 16:12:07.751755 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.744012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:49881 (local address 127.18.90.62:34731)
0806 16:12:07.744133 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.744138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.744407 (+ 269us) server_negotiation.cc:407] Connection header received
0806 16:12:07.744560 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.744563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.744610 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.744692 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.745135 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.745787 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.746986 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.747214 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.747321 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.748510 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.748513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.748516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.748540 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.750044 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.750458 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.750464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.750466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.750562 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.750829 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.750835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.750839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.751077 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.751182 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.751397 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.751528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:12:07 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:07.774814 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.767005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.18.90.62:34731)
0806 16:12:07.767136 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.767140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.767153 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:07.767202 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.767204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.767245 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.767318 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:07.768051 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.768664 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.769445 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.769652 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.769733 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.772040 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:07.772043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:07.772045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:07.772070 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:07.773363 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.773727 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.773730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.773731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.773804 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:07.774090 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:07.774092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:07.774094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:07.774279 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:07.774379 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.774586 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.774686 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:12:07.777957 19926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:07.781108 19926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:07.796559 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.790357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38187 (local address 127.0.0.1:40488)
0806 16:12:07.791817 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.791832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.791925 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.792436 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.792438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.792457 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:07.792569 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.792576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.793877 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.793883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.794829 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.794839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.795954 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.795958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.796065 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.796113 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.796220 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.796372 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260806 16:12:07.796736 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.790414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.18.90.1:43113)
0806 16:12:07.790735 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.790743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.791424 (+ 681us) server_negotiation.cc:407] Connection header received
0806 16:12:07.791511 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.791517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.791785 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.791937 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:07.792464 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.793632 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.795296 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.796072 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.796179 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.796359 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.796483 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.796578 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260806 16:12:07.796721 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.790301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43113 (local address 127.0.0.1:35770)
0806 16:12:07.791082 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.791131 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.791276 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.792126 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.792132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.792155 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:07.792329 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.792339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.793765 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.793768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.795163 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.795173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.796200 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.796204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.796334 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.796358 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.796443 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.796512 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":27,"spinlock_wait_cycles":2432,"thread_start_us":64,"threads_started":1}
I20260806 16:12:07.797073 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.791350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.18.90.3:44853)
0806 16:12:07.792486 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.792498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.792520 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:12:07.792592 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.792599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.792776 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.792923 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:07.793122 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.793956 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.795350 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.796084 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.796206 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.796402 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.796841 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.796936 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":123,"threads_started":1}
I20260806 16:12:07.796856 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.791183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44853 (local address 127.0.0.1:41972)
0806 16:12:07.792112 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.792125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.792210 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.792913 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.792916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.792934 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:07.793011 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.793018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.794089 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.794092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.795209 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.795220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.796463 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.796467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.796546 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.796567 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.796638 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.796695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":3,"spinlock_wait_cycles":2688,"thread_start_us":114,"threads_started":1}
I20260806 16:12:07.798105 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.791110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.18.90.2:38187)
0806 16:12:07.791565 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.791576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.791966 (+ 390us) server_negotiation.cc:407] Connection header received
0806 16:12:07.792072 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.792079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.792289 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.792449 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:07.792681 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.793771 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.794958 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.795677 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.795781 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.797842 (+ 2061us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.797891 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.797960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":152,"threads_started":1}
I20260806 16:12:07.799882 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae1468135240ac96b5d4265c88a8fe (DEFAULT_TABLE table=test-table [id=261d5f789f6d4acc811dd42f12583b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:07.799959 20064 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae1468135240ac96b5d4265c88a8fe (DEFAULT_TABLE table=test-table [id=261d5f789f6d4acc811dd42f12583b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:07.800459 20205 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae1468135240ac96b5d4265c88a8fe (DEFAULT_TABLE table=test-table [id=261d5f789f6d4acc811dd42f12583b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:07.800985 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae1468135240ac96b5d4265c88a8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:07.800998 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae1468135240ac96b5d4265c88a8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:07.801304 20205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae1468135240ac96b5d4265c88a8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:07.807566 20442 tablet_bootstrap.cc:492] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Bootstrap starting.
I20260806 16:12:07.808625 20443 tablet_bootstrap.cc:492] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Bootstrap starting.
I20260806 16:12:07.810034 20442 tablet_bootstrap.cc:654] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:07.810564 20443 tablet_bootstrap.cc:654] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:07.810815 20442 log.cc:824] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:07.811326 20443 log.cc:824] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:07.811437 20444 tablet_bootstrap.cc:492] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Bootstrap starting.
I20260806 16:12:07.813120 20444 tablet_bootstrap.cc:654] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:07.813318 20443 tablet_bootstrap.cc:492] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: No bootstrap required, opened a new log
I20260806 16:12:07.813437 20442 tablet_bootstrap.cc:492] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: No bootstrap required, opened a new log
I20260806 16:12:07.813515 20443 ts_tablet_manager.cc:1397] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:12:07.813585 20442 ts_tablet_manager.cc:1397] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:12:07.813931 20444 log.cc:824] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:07.815742 20444 tablet_bootstrap.cc:492] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: No bootstrap required, opened a new log
I20260806 16:12:07.815932 20444 ts_tablet_manager.cc:1397] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 16:12:07.817693 20442 raft_consensus.cc:348] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.817974 20442 raft_consensus.cc:374] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:07.818070 20442 raft_consensus.cc:729] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e1c3df8f7a4529b68b75caa900ea57, State: Initialized, Role: FOLLOWER
I20260806 16:12:07.818186 20443 raft_consensus.cc:348] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.818416 20443 raft_consensus.cc:374] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:07.818500 20443 raft_consensus.cc:729] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb706bea8114528b58b37d9dc2d6126, State: Initialized, Role: FOLLOWER
I20260806 16:12:07.818549 20442 consensus_queue.cc:260] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.818975 20443 consensus_queue.cc:260] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.819245 20418 heartbeater.cc:503] Master 127.18.90.62:34731 was elected leader, sending a full tablet report...
I20260806 16:12:07.819118 20444 raft_consensus.cc:348] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.819433 20444 raft_consensus.cc:374] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:07.819530 20444 raft_consensus.cc:729] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8699b62042454890045979d5fe22ec, State: Initialized, Role: FOLLOWER
I20260806 16:12:07.819584 20442 ts_tablet_manager.cc:1428] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:12:07.819969 20443 ts_tablet_manager.cc:1428] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:12:07.820035 20131 heartbeater.cc:503] Master 127.18.90.62:34731 was elected leader, sending a full tablet report...
I20260806 16:12:07.819989 20444 consensus_queue.cc:260] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.820796 20274 heartbeater.cc:503] Master 127.18.90.62:34731 was elected leader, sending a full tablet report...
I20260806 16:12:07.821091 20444 ts_tablet_manager.cc:1428] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260806 16:12:07.948627 20449 raft_consensus.cc:482] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:07.949136 20449 raft_consensus.cc:504] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.950387 20449 leader_election.cc:288] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187), 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853)
I20260806 16:12:07.954399 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.950856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:49611 (local address 127.18.90.3:44853)
0806 16:12:07.950989 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.950994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.951159 (+ 165us) server_negotiation.cc:407] Connection header received
0806 16:12:07.951360 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.951365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.951415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.951502 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:07.951784 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.952332 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.953411 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.953808 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.953925 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.953992 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.954183 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.954252 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:12:07.954463 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.950768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:54407 (local address 127.18.90.2:38187)
0806 16:12:07.950908 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.950913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.951177 (+ 264us) server_negotiation.cc:407] Connection header received
0806 16:12:07.951373 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.951378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.951430 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.951515 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:07.951806 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.952403 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.953451 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.953864 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.953986 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.954050 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.954257 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.954334 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:12:07.954478 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.950743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44853 (local address 127.18.90.1:49611)
0806 16:12:07.951064 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.951076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.951250 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.951516 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.951519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.951535 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:07.951668 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.951674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.952450 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.952453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.953266 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.953275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.953931 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.953935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.954031 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.954062 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.954167 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.954248 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":4,"spinlock_wait_cycles":29824,"thread_start_us":126,"threads_started":1}
I20260806 16:12:07.954486 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.950592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38187 (local address 127.18.90.1:54407)
0806 16:12:07.951062 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:07.951076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:07.951252 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:07.951528 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:07.951531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:07.951546 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:07.951681 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.951687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.952516 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.952521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.953318 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:07.953326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.953990 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.953993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:07.954123 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.954141 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:12:07.954230 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:12:07.954302 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"spinlock_wait_cycles":10368,"thread_start_us":154,"threads_started":1}
I20260806 16:12:07.955222 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae1468135240ac96b5d4265c88a8fe" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e1c3df8f7a4529b68b75caa900ea57" is_pre_election: true
I20260806 16:12:07.955240 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae1468135240ac96b5d4265c88a8fe" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8699b62042454890045979d5fe22ec" is_pre_election: true
I20260806 16:12:07.955542 20228 raft_consensus.cc:2454] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 0.
I20260806 16:12:07.955545 20371 raft_consensus.cc:2454] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 0.
I20260806 16:12:07.956037 20017 leader_election.cc:302] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8699b62042454890045979d5fe22ec, 9bb706bea8114528b58b37d9dc2d6126; no voters:
I20260806 16:12:07.956362 20449 raft_consensus.cc:2781] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:07.956454 20449 raft_consensus.cc:482] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:07.956506 20449 raft_consensus.cc:3037] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:07.957571 20449 raft_consensus.cc:504] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.957964 20449 leader_election.cc:288] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187), 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853)
I20260806 16:12:07.958312 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae1468135240ac96b5d4265c88a8fe" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8699b62042454890045979d5fe22ec"
I20260806 16:12:07.958312 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae1468135240ac96b5d4265c88a8fe" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e1c3df8f7a4529b68b75caa900ea57"
I20260806 16:12:07.958443 20371 raft_consensus.cc:3037] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:07.958465 20228 raft_consensus.cc:3037] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:07.960186 20371 raft_consensus.cc:2454] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 1.
I20260806 16:12:07.960309 20228 raft_consensus.cc:2454] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 1.
I20260806 16:12:07.960556 20014 leader_election.cc:302] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e1c3df8f7a4529b68b75caa900ea57, 9bb706bea8114528b58b37d9dc2d6126; no voters:
I20260806 16:12:07.960785 20449 raft_consensus.cc:2781] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:07.961012 20449 raft_consensus.cc:686] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 1 LEADER]: Becoming Leader. State: Replica: 9bb706bea8114528b58b37d9dc2d6126, State: Running, Role: LEADER
I20260806 16:12:07.961364 20449 consensus_queue.cc:237] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:07.964759 19926 catalog_manager.cc:5495] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1). New cstate: current_term: 1 leader_uuid: "9bb706bea8114528b58b37d9dc2d6126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } health_report { overall_health: UNKNOWN } } }
W20260806 16:12:07.987980 20419 tablet.cc:2369] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:07.993896 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:07.990265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.18.90.1:43113)
0806 16:12:07.990422 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:07.990427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:07.990442 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:07.990519 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:07.990521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:07.990574 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:07.990673 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:07.990922 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.991424 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.992324 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:07.992555 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:07.992650 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:07.992989 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:07.993425 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:07.993517 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:07.993706 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 16:12:07.993783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 16:12:07.999846 20228 raft_consensus.cc:1261] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Refusing update from remote peer 9bb706bea8114528b58b37d9dc2d6126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:07.999847 20371 raft_consensus.cc:1261] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Refusing update from remote peer 9bb706bea8114528b58b37d9dc2d6126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:08.000567 20449 consensus_queue.cc:1035] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:08.000785 20453 consensus_queue.cc:1035] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:08.009639 20457 mvcc.cc:205] Tried to move back new op lower bound from 7315590053879599104 to 7315590053732745216. Current Snapshot: MvccSnapshot[applied={T|T < 7315590053879599104}]
I20260806 16:12:08.010675 20458 mvcc.cc:205] Tried to move back new op lower bound from 7315590053879599104 to 7315590053732745216. Current Snapshot: MvccSnapshot[applied={T|T < 7315590053879599104}]
I20260806 16:12:08.020916 19926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:08.021107 19926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:08.023545 19926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: Sending DeleteTablet for 3 replicas of tablet 6eae1468135240ac96b5d4265c88a8fe
I20260806 16:12:08.024374 20064 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eae1468135240ac96b5d4265c88a8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:08 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:35770
I20260806 16:12:08.024389 20205 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eae1468135240ac96b5d4265c88a8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:08 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:40488
I20260806 16:12:08.024961 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eae1468135240ac96b5d4265c88a8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:08 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:41972
I20260806 16:12:08.025295 20467 tablet_replica.cc:330] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: stopping tablet replica
I20260806 16:12:08.025439 19926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 16:12:08.025552 20468 tablet_replica.cc:330] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: stopping tablet replica
I20260806 16:12:08.025753 20467 raft_consensus.cc:2229] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 1 LEADER]: Raft consensus shutting down.
W20260806 16:12:08.025868 19926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:08.026013 20468 raft_consensus.cc:2229] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:08.026238 20467 raft_consensus.cc:2258] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:08.026448 20468 raft_consensus.cc:2258] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:08.026744 20466 tablet_replica.cc:330] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: stopping tablet replica
I20260806 16:12:08.027293 20466 raft_consensus.cc:2229] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:08.027619 20466 raft_consensus.cc:2258] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:08.027863 20467 ts_tablet_manager.cc:1905] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:08.029066 20466 ts_tablet_manager.cc:1905] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:08.030903 20467 ts_tablet_manager.cc:1918] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:08.031060 20467 log.cc:1196] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/6eae1468135240ac96b5d4265c88a8fe
I20260806 16:12:08.031497 20467 ts_tablet_manager.cc:1939] T 6eae1468135240ac96b5d4265c88a8fe P 9bb706bea8114528b58b37d9dc2d6126: Deleting consensus metadata
I20260806 16:12:08.031791 20205 tablet_service.cc:1467] Processing CreateTablet for tablet 8b72223d2bb3412a9f18438d8af97eca (DEFAULT_TABLE table=test-table [id=77b0f6bb1c4a47fab05c47ad26499c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:08.031980 20466 ts_tablet_manager.cc:1918] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:08.032058 20205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b72223d2bb3412a9f18438d8af97eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:08.032128 20466 log.cc:1196] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/6eae1468135240ac96b5d4265c88a8fe
I20260806 16:12:08.032501 20466 ts_tablet_manager.cc:1939] T 6eae1468135240ac96b5d4265c88a8fe P 8b8699b62042454890045979d5fe22ec: Deleting consensus metadata
I20260806 16:12:08.034040 20468 ts_tablet_manager.cc:1905] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:08.034266 19911 catalog_manager.cc:4841] TS 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1:43113): tablet 6eae1468135240ac96b5d4265c88a8fe (table test-table [id=261d5f789f6d4acc811dd42f12583b01]) successfully deleted
I20260806 16:12:08.034446 20444 tablet_bootstrap.cc:492] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Bootstrap starting.
I20260806 16:12:08.035156 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 8b72223d2bb3412a9f18438d8af97eca (DEFAULT_TABLE table=test-table [id=77b0f6bb1c4a47fab05c47ad26499c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:08.035441 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b72223d2bb3412a9f18438d8af97eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:08.035465 20444 tablet_bootstrap.cc:654] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:08.036150 19913 catalog_manager.cc:4841] TS 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187): tablet 6eae1468135240ac96b5d4265c88a8fe (table test-table [id=261d5f789f6d4acc811dd42f12583b01]) successfully deleted
I20260806 16:12:08.036748 20064 tablet_service.cc:1467] Processing CreateTablet for tablet 8b72223d2bb3412a9f18438d8af97eca (DEFAULT_TABLE table=test-table [id=77b0f6bb1c4a47fab05c47ad26499c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:08.037012 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b72223d2bb3412a9f18438d8af97eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:08.037910 20442 tablet_bootstrap.cc:492] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Bootstrap starting.
I20260806 16:12:08.038914 20442 tablet_bootstrap.cc:654] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:08.037895 20468 ts_tablet_manager.cc:1918] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:08.039708 20468 log.cc:1196] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/6eae1468135240ac96b5d4265c88a8fe
I20260806 16:12:08.039813 20444 tablet_bootstrap.cc:492] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: No bootstrap required, opened a new log
I20260806 16:12:08.039913 20444 ts_tablet_manager.cc:1397] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 16:12:08.040130 20442 tablet_bootstrap.cc:492] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: No bootstrap required, opened a new log
I20260806 16:12:08.040227 20442 ts_tablet_manager.cc:1397] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:08.040324 20468 ts_tablet_manager.cc:1939] T 6eae1468135240ac96b5d4265c88a8fe P 12e1c3df8f7a4529b68b75caa900ea57: Deleting consensus metadata
I20260806 16:12:08.040452 20444 raft_consensus.cc:348] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.040572 20444 raft_consensus.cc:374] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:08.040607 20444 raft_consensus.cc:729] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8699b62042454890045979d5fe22ec, State: Initialized, Role: FOLLOWER
I20260806 16:12:08.040699 20442 raft_consensus.cc:348] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.040799 20442 raft_consensus.cc:374] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:08.040831 20442 raft_consensus.cc:729] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e1c3df8f7a4529b68b75caa900ea57, State: Initialized, Role: FOLLOWER
I20260806 16:12:08.040786 20444 consensus_queue.cc:260] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.040949 20442 consensus_queue.cc:260] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.041226 20442 ts_tablet_manager.cc:1428] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:08.041594 20444 ts_tablet_manager.cc:1428] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:08.041843 20443 tablet_bootstrap.cc:492] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Bootstrap starting.
I20260806 16:12:08.042913 20443 tablet_bootstrap.cc:654] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:08.044132 19910 catalog_manager.cc:4841] TS 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853): tablet 6eae1468135240ac96b5d4265c88a8fe (table test-table [id=261d5f789f6d4acc811dd42f12583b01]) successfully deleted
I20260806 16:12:08.044751 20443 tablet_bootstrap.cc:492] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: No bootstrap required, opened a new log
I20260806 16:12:08.044845 20443 ts_tablet_manager.cc:1397] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:08.045291 20443 raft_consensus.cc:348] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.045390 20443 raft_consensus.cc:374] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:08.045413 20443 raft_consensus.cc:729] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb706bea8114528b58b37d9dc2d6126, State: Initialized, Role: FOLLOWER
I20260806 16:12:08.045559 20443 consensus_queue.cc:260] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.045840 20443 ts_tablet_manager.cc:1428] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:12:08.071926 20461 raft_consensus.cc:482] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:08.072063 20461 raft_consensus.cc:504] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.072584 20461 leader_election.cc:288] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187), 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853)
I20260806 16:12:08.073035 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b72223d2bb3412a9f18438d8af97eca" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e1c3df8f7a4529b68b75caa900ea57" is_pre_election: true
I20260806 16:12:08.073053 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b72223d2bb3412a9f18438d8af97eca" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8699b62042454890045979d5fe22ec" is_pre_election: true
I20260806 16:12:08.073201 20228 raft_consensus.cc:2454] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 0.
I20260806 16:12:08.073215 20371 raft_consensus.cc:2454] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 0.
I20260806 16:12:08.073536 20017 leader_election.cc:302] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8699b62042454890045979d5fe22ec, 9bb706bea8114528b58b37d9dc2d6126; no voters:
I20260806 16:12:08.073694 20461 raft_consensus.cc:2781] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:08.073757 20461 raft_consensus.cc:482] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:08.073810 20461 raft_consensus.cc:3037] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:08.074838 20461 raft_consensus.cc:504] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.075215 20461 leader_election.cc:288] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187), 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853)
I20260806 16:12:08.075546 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b72223d2bb3412a9f18438d8af97eca" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8699b62042454890045979d5fe22ec"
I20260806 16:12:08.075567 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b72223d2bb3412a9f18438d8af97eca" candidate_uuid: "9bb706bea8114528b58b37d9dc2d6126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e1c3df8f7a4529b68b75caa900ea57"
I20260806 16:12:08.075675 20228 raft_consensus.cc:3037] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:08.075707 20371 raft_consensus.cc:3037] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:08.076807 20371 raft_consensus.cc:2454] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 1.
I20260806 16:12:08.076869 20228 raft_consensus.cc:2454] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb706bea8114528b58b37d9dc2d6126 in term 1.
I20260806 16:12:08.077108 20014 leader_election.cc:302] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e1c3df8f7a4529b68b75caa900ea57, 9bb706bea8114528b58b37d9dc2d6126; no voters:
I20260806 16:12:08.077270 20461 raft_consensus.cc:2781] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:08.077365 20461 raft_consensus.cc:686] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 1 LEADER]: Becoming Leader. State: Replica: 9bb706bea8114528b58b37d9dc2d6126, State: Running, Role: LEADER
I20260806 16:12:08.077503 20461 consensus_queue.cc:237] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.079327 19925 catalog_manager.cc:5495] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1). New cstate: current_term: 1 leader_uuid: "9bb706bea8114528b58b37d9dc2d6126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } health_report { overall_health: UNKNOWN } } }
Aug 06 16:12:08 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032726, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:08.127519 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.117952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34731 (local address 127.0.0.1:38676)
0806 16:12:08.118304 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:08.118325 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:08.118431 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:08.118703 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:08.118705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:08.119066 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:08.119166 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.119193 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.120345 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.120348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.120923 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.120931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.121498 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.121500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.121608 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.122283 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:08.122295 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:08.123753 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:08.125828 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:08.125838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:08.125843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:08.126090 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:08.126447 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:08.126450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:08.126451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:08.126534 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:08.126994 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:08.127000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:08.127124 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 16:12:08.127250 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 16:12:08.127376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":31,"thread_start_us":114,"threads_started":1}
I20260806 16:12:08.127669 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.118071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.18.90.62:34731)
0806 16:12:08.118233 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.118238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.118398 (+ 160us) server_negotiation.cc:407] Connection header received
0806 16:12:08.118518 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.118521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.118572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.118670 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:08.119304 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.120129 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.121051 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.121375 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.121485 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.123934 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:08.123937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:08.123940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:08.123967 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:08.125685 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:08.126248 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:08.126253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:08.126256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:08.126352 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:08.126620 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:08.126623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:08.126625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:08.126865 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:08.126968 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.127334 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.127462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":55}
I20260806 16:12:08.131986 19925 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:38676:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 16:12:08.132874 19925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:08.138159 20064 tablet_service.cc:1467] Processing CreateTablet for tablet 9f60aeab499a462dbd3c059ccd6d2633 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2d06325d8a24f288b4dfcabadd97c8e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:08.138312 20205 tablet_service.cc:1467] Processing CreateTablet for tablet 9f60aeab499a462dbd3c059ccd6d2633 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2d06325d8a24f288b4dfcabadd97c8e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:08.138576 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f60aeab499a462dbd3c059ccd6d2633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:08.138736 20205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f60aeab499a462dbd3c059ccd6d2633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:08.141444 20443 tablet_bootstrap.cc:492] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: Bootstrap starting.
I20260806 16:12:08.142376 20444 tablet_bootstrap.cc:492] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: Bootstrap starting.
I20260806 16:12:08.142712 20443 tablet_bootstrap.cc:654] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:08.142709 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 9f60aeab499a462dbd3c059ccd6d2633 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2d06325d8a24f288b4dfcabadd97c8e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:08.143133 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f60aeab499a462dbd3c059ccd6d2633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:08.143432 20444 tablet_bootstrap.cc:654] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:08.144897 20444 tablet_bootstrap.cc:492] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: No bootstrap required, opened a new log
I20260806 16:12:08.144999 20444 ts_tablet_manager.cc:1397] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:08.145485 20444 raft_consensus.cc:348] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.145619 20444 raft_consensus.cc:374] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:08.145654 20444 raft_consensus.cc:729] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8699b62042454890045979d5fe22ec, State: Initialized, Role: FOLLOWER
I20260806 16:12:08.145787 20444 consensus_queue.cc:260] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.146062 20442 tablet_bootstrap.cc:492] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: Bootstrap starting.
I20260806 16:12:08.146085 20450 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.146255 20450 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:08.147151 20442 tablet_bootstrap.cc:654] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:08.147989 20444 ts_tablet_manager.cc:1428] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:08.148957 20442 tablet_bootstrap.cc:492] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: No bootstrap required, opened a new log
I20260806 16:12:08.149053 20442 ts_tablet_manager.cc:1397] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:08.149513 20442 raft_consensus.cc:348] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.149670 20442 raft_consensus.cc:374] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:08.149717 20442 raft_consensus.cc:729] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e1c3df8f7a4529b68b75caa900ea57, State: Initialized, Role: FOLLOWER
I20260806 16:12:08.149843 20442 consensus_queue.cc:260] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.150100 20448 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.150281 20448 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:08.150528 20442 ts_tablet_manager.cc:1428] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:08.153209 20443 tablet_bootstrap.cc:492] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: No bootstrap required, opened a new log
I20260806 16:12:08.153334 20443 ts_tablet_manager.cc:1397] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260806 16:12:08.153774 20443 raft_consensus.cc:348] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.153898 20443 raft_consensus.cc:374] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:08.153939 20443 raft_consensus.cc:729] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb706bea8114528b58b37d9dc2d6126, State: Initialized, Role: FOLLOWER
I20260806 16:12:08.154098 20443 consensus_queue.cc:260] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.154412 20443 ts_tablet_manager.cc:1428] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:08.154385 20449 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.154523 20449 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 16:12:08.261468 20275 tablet.cc:2369] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:12:08.274958 20132 tablet.cc:2369] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:08.357743 20448 raft_consensus.cc:482] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:08.357918 20448 raft_consensus.cc:504] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.359139 20448 leader_election.cc:288] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1:43113), 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187)
I20260806 16:12:08.363355 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.359592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:49461 (local address 127.18.90.2:38187)
0806 16:12:08.359859 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.359865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.359888 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:12:08.360073 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.360077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.360133 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.360193 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:08.360631 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.361176 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.362344 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.362749 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.362864 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.362932 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.363128 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.363211 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260806 16:12:08.363379 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.359694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:36651 (local address 127.18.90.1:43113)
0806 16:12:08.359901 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.359905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.359927 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:12:08.360124 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.360126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.360174 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.360438 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:08.360709 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.361196 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.362351 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.362755 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.362877 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.362940 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.363187 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.363275 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 16:12:08.363379 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.359470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38187 (local address 127.18.90.3:49461)
0806 16:12:08.359796 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:08.359816 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:08.359937 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:08.360327 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:08.360331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:08.360352 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:08.360503 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.360513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.361323 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.361327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.362210 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.362219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.362880 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.362883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.362984 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.363024 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 16:12:08.363084 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 16:12:08.363174 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":22,"thread_start_us":82,"threads_started":1}
I20260806 16:12:08.363384 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.359470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43113 (local address 127.18.90.3:36651)
0806 16:12:08.359857 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:08.359872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:08.359977 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:08.360453 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:08.360455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:08.360470 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:08.360593 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.360599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.361350 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.361353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.362219 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.362228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.362901 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.362905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.363033 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.363050 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:12:08.363149 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:12:08.363238 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":130,"thread_start_us":95,"threads_started":1}
I20260806 16:12:08.364059 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f60aeab499a462dbd3c059ccd6d2633" candidate_uuid: "12e1c3df8f7a4529b68b75caa900ea57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8699b62042454890045979d5fe22ec" is_pre_election: true
I20260806 16:12:08.364245 20228 raft_consensus.cc:2454] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e1c3df8f7a4529b68b75caa900ea57 in term 0.
I20260806 16:12:08.364225 20085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f60aeab499a462dbd3c059ccd6d2633" candidate_uuid: "12e1c3df8f7a4529b68b75caa900ea57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb706bea8114528b58b37d9dc2d6126" is_pre_election: true
I20260806 16:12:08.364482 20085 raft_consensus.cc:2454] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e1c3df8f7a4529b68b75caa900ea57 in term 0.
I20260806 16:12:08.364667 20305 leader_election.cc:302] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e1c3df8f7a4529b68b75caa900ea57, 8b8699b62042454890045979d5fe22ec; no voters:
I20260806 16:12:08.364962 20448 raft_consensus.cc:2781] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:08.365037 20448 raft_consensus.cc:482] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:08.365097 20448 raft_consensus.cc:3037] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:08.366211 20448 raft_consensus.cc:504] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.366583 20448 leader_election.cc:288] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [CANDIDATE]: Term 1 election: Requested vote from peers 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1:43113), 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187)
I20260806 16:12:08.366976 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f60aeab499a462dbd3c059ccd6d2633" candidate_uuid: "12e1c3df8f7a4529b68b75caa900ea57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8699b62042454890045979d5fe22ec"
I20260806 16:12:08.366976 20085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f60aeab499a462dbd3c059ccd6d2633" candidate_uuid: "12e1c3df8f7a4529b68b75caa900ea57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb706bea8114528b58b37d9dc2d6126"
I20260806 16:12:08.367100 20228 raft_consensus.cc:3037] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:08.367100 20085 raft_consensus.cc:3037] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:08.368300 20228 raft_consensus.cc:2454] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e1c3df8f7a4529b68b75caa900ea57 in term 1.
I20260806 16:12:08.368330 20085 raft_consensus.cc:2454] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e1c3df8f7a4529b68b75caa900ea57 in term 1.
I20260806 16:12:08.368664 20303 leader_election.cc:302] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12e1c3df8f7a4529b68b75caa900ea57, 8b8699b62042454890045979d5fe22ec, 9bb706bea8114528b58b37d9dc2d6126; no voters:
I20260806 16:12:08.368862 20448 raft_consensus.cc:2781] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:08.369071 20448 raft_consensus.cc:686] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 LEADER]: Becoming Leader. State: Replica: 12e1c3df8f7a4529b68b75caa900ea57, State: Running, Role: LEADER
I20260806 16:12:08.369334 20448 consensus_queue.cc:237] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } }
I20260806 16:12:08.370616 20484 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: TxnStatusTablet state changed. Reason: New leader 12e1c3df8f7a4529b68b75caa900ea57. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.370730 20484 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:08.371137 20487 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 16:12:08.371313 20487 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 16:12:08.372016 19925 catalog_manager.cc:5495] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:08.390617 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.386453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.18.90.3:44853)
0806 16:12:08.386592 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.386596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.386681 (+ 85us) server_negotiation.cc:407] Connection header received
0806 16:12:08.386893 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.386896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.386952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.387029 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:08.387300 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.387842 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.389126 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.389358 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.389441 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.389787 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:08.390246 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:08.390332 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.390450 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.390502 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:08.390674 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.386342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44853 (local address 127.0.0.1:41986)
0806 16:12:08.386600 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:08.386617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:08.386743 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:08.387069 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:08.387072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:08.387080 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:08.387195 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.387201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.387964 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.387967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.388992 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.389004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.389471 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.389475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.389576 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.389633 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:08.390340 (+ 707us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:08.390355 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 16:12:08.390429 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 16:12:08.390525 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 16:12:08.394212 20228 raft_consensus.cc:1261] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Refusing update from remote peer 12e1c3df8f7a4529b68b75caa900ea57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:08.394263 20085 raft_consensus.cc:1261] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Refusing update from remote peer 12e1c3df8f7a4529b68b75caa900ea57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:08.394848 20484 consensus_queue.cc:1035] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:08.395032 20448 consensus_queue.cc:1035] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:08.396920 20450 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: TxnStatusTablet state changed. Reason: New leader 12e1c3df8f7a4529b68b75caa900ea57. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.397045 20450 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:08.397194 20449 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: TxnStatusTablet state changed. Reason: New leader 12e1c3df8f7a4529b68b75caa900ea57. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.397300 20449 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:08.398795 20448 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.398933 20448 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:08.399819 20450 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.399938 20450 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 8b8699b62042454890045979d5fe22ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:08.400780 20484 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.400874 20484 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 12e1c3df8f7a4529b68b75caa900ea57: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:08.402832 20449 tablet_replica.cc:439] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12e1c3df8f7a4529b68b75caa900ea57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb706bea8114528b58b37d9dc2d6126" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43113 } } peers { permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 } } peers { permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 } } }
I20260806 16:12:08.402946 20449 tablet_replica.cc:442] T 9f60aeab499a462dbd3c059ccd6d2633 P 9bb706bea8114528b58b37d9dc2d6126: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:08.414963 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.411449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:51287 (local address 127.18.90.3:44853)
0806 16:12:08.411578 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.411584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.411771 (+ 187us) server_negotiation.cc:407] Connection header received
0806 16:12:08.411933 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.411938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.411990 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.412072 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:08.412368 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.412877 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.413831 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.414066 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.414178 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.414483 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:08.414554 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:08.414634 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.414745 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.414797 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:12:08.414963 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.411344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44853 (local address 127.18.90.1:51287)
0806 16:12:08.411679 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:08.411692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:08.411824 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:08.412089 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:08.412092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:08.412101 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:08.412217 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.412223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.412998 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.413001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.413678 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.413688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.414185 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.414189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.414278 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.414333 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:08.414647 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:08.414656 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:12:08.414733 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:12:08.414825 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260806 16:12:08.421921 20228 raft_consensus.cc:1261] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Refusing update from remote peer 9bb706bea8114528b58b37d9dc2d6126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:08.421919 20371 raft_consensus.cc:1261] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Refusing update from remote peer 9bb706bea8114528b58b37d9dc2d6126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:08.422283 20449 consensus_queue.cc:1035] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e1c3df8f7a4529b68b75caa900ea57" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:08.422420 20461 consensus_queue.cc:1035] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8699b62042454890045979d5fe22ec" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:08.426098 20456 mvcc.cc:205] Tried to move back new op lower bound from 7315590055612792832 to 7315590054207066112. Current Snapshot: MvccSnapshot[applied={T|T < 7315590055612792832}]
I20260806 16:12:08.445007 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.440485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43113 (local address 127.18.90.3:53391)
0806 16:12:08.441027 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:08.441044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:08.441156 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:08.441524 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:08.441530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:08.441539 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:08.441680 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.441688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.442940 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.442942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.443679 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:08.443689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.444183 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.444187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:08.444318 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.444375 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:08.444689 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:08.444697 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:12:08.444763 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 16:12:08.444846 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":120,"threads_started":1}
I20260806 16:12:08.444986 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.440533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:53391 (local address 127.18.90.1:43113)
0806 16:12:08.440660 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.440665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.441107 (+ 442us) server_negotiation.cc:407] Connection header received
0806 16:12:08.441270 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.441274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.441333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.441401 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:08.441859 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.442573 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.443802 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.444044 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.444161 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.444516 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:08.444601 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:08.444682 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.444774 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.444832 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:12:08.470887 19923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:08.471076 19923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:08.473158 19923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eaded62a40d40088a5ebc5a7f28e546: Sending DeleteTablet for 3 replicas of tablet 8b72223d2bb3412a9f18438d8af97eca
I20260806 16:12:08.473752 20064 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b72223d2bb3412a9f18438d8af97eca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:08 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:35770
I20260806 16:12:08.473870 20205 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b72223d2bb3412a9f18438d8af97eca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:08 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:40488
I20260806 16:12:08.474077 20467 tablet_replica.cc:330] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: stopping tablet replica
I20260806 16:12:08.474203 20467 raft_consensus.cc:2229] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:08.474206 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b72223d2bb3412a9f18438d8af97eca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:08 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:41972
I20260806 16:12:08.474402 20467 raft_consensus.cc:2258] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:08.474408 20468 tablet_replica.cc:330] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: stopping tablet replica
I20260806 16:12:08.474540 20468 raft_consensus.cc:2229] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:08.474581 20466 tablet_replica.cc:330] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: stopping tablet replica
I20260806 16:12:08.474686 20468 raft_consensus.cc:2258] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:08.474689 20466 raft_consensus.cc:2229] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:08.474814 20466 raft_consensus.cc:2258] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:08.475680 20468 ts_tablet_manager.cc:1905] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:08.475698 20467 ts_tablet_manager.cc:1905] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:08.478670 20467 ts_tablet_manager.cc:1918] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:08.478751 20467 log.cc:1196] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/8b72223d2bb3412a9f18438d8af97eca
I20260806 16:12:08.478905 20468 ts_tablet_manager.cc:1918] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:08.478976 20468 log.cc:1196] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/8b72223d2bb3412a9f18438d8af97eca
I20260806 16:12:08.479031 20467 ts_tablet_manager.cc:1939] T 8b72223d2bb3412a9f18438d8af97eca P 9bb706bea8114528b58b37d9dc2d6126: Deleting consensus metadata
I20260806 16:12:08.479229 20468 ts_tablet_manager.cc:1939] T 8b72223d2bb3412a9f18438d8af97eca P 12e1c3df8f7a4529b68b75caa900ea57: Deleting consensus metadata
I20260806 16:12:08.479737 19911 catalog_manager.cc:4841] TS 9bb706bea8114528b58b37d9dc2d6126 (127.18.90.1:43113): tablet 8b72223d2bb3412a9f18438d8af97eca (table test-table [id=77b0f6bb1c4a47fab05c47ad26499c3e]) successfully deleted
I20260806 16:12:08.479924 19910 catalog_manager.cc:4841] TS 12e1c3df8f7a4529b68b75caa900ea57 (127.18.90.3:44853): tablet 8b72223d2bb3412a9f18438d8af97eca (table test-table [id=77b0f6bb1c4a47fab05c47ad26499c3e]) successfully deleted
I20260806 16:12:08.480587 20466 ts_tablet_manager.cc:1905] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:08.483299 20466 ts_tablet_manager.cc:1918] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:08.483382 20466 log.cc:1196] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/8b72223d2bb3412a9f18438d8af97eca
Aug 06 16:12:08 dist-test-slave-0b58 krb5kdc[19881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:08.483652 20466 ts_tablet_manager.cc:1939] T 8b72223d2bb3412a9f18438d8af97eca P 8b8699b62042454890045979d5fe22ec: Deleting consensus metadata
I20260806 16:12:08.484468 19913 catalog_manager.cc:4841] TS 8b8699b62042454890045979d5fe22ec (127.18.90.2:38187): tablet 8b72223d2bb3412a9f18438d8af97eca (table test-table [id=77b0f6bb1c4a47fab05c47ad26499c3e]) successfully deleted
I20260806 16:12:08.488093 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.476348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.18.90.1:43113)
0806 16:12:08.476470 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.476475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.477027 (+ 552us) server_negotiation.cc:407] Connection header received
0806 16:12:08.477228 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.477231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.477285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.477337 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:08.478423 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.479246 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.480454 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.480738 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.480835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.484212 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:08.484226 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:08.484243 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:08.484318 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:08.486287 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:08.486747 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:08.486753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:08.486754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:08.486831 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:08.487116 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:08.487119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:08.487121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:08.487379 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:08.487602 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.487780 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.487921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 16:12:08.488636 20125 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35786
I20260806 16:12:08.499339 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:08.490422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.18.90.62:34731)
0806 16:12:08.490556 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:08.490561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:08.490795 (+ 234us) server_negotiation.cc:407] Connection header received
0806 16:12:08.491009 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:08.491013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:08.491073 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:08.491145 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:08.491994 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.492767 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.493968 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:08.494289 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:08.494385 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:08.496023 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:08.496026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:08.496028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:08.496056 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:08.497643 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:08.498133 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:08.498135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:08.498137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:08.498215 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:08.498496 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:08.498499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:08.498500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:08.498681 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:08.498783 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:08.499048 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 16:12:08.499150 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 16:12:08.499848 19923 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38682
I20260806 16:12:08.501157 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19998
I20260806 16:12:08.513464 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 20140
I20260806 16:12:08.522607 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 20284
I20260806 16:12:08.530596 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 19894
2026-08-06T16:12:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2952 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:08 dist-test-slave-0b58 krb5kdc[20508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:08 dist-test-slave-0b58 krb5kdc[20508](info): set up 2 sockets
Aug 06 16:12:08 dist-test-slave-0b58 krb5kdc[20508](info): commencing operation
krb5kdc: starting...
W20260806 16:12:09.588948 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:09 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:09.713138 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33753
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:09.827488 20521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:09.827797 20521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:09.827842 20521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:09.832028 20521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:09.832096 20521 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 16:12:09.832113 20521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:09.832166 20521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:09.832199 20521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:09.832221 20521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:09.837626 20521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33753
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:09.838785 20521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:09.839874 20521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:09.848568 20527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:09.848588 20529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:09.848639 20521 server_base.cc:1034] running on GCE node
W20260806 16:12:09.848615 20526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:09.849474 20521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:09.850653 20521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:09.851845 20521 hybrid_clock.cc:648] HybridClock initialized: now 1786032729851825 us; error 32 us; skew 500 ppm
Aug 06 16:12:09 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032729, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:09.854655 20521 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:09.855559 20521 webserver.cc:460] Webserver started at http://127.18.90.62:42335/ using document root <none> and password file <none>
I20260806 16:12:09.856137 20521 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:09.856235 20521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:09.856525 20521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:09.858685 20521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "a6d487d6b52949bf98fc4d6c1f19c98c"
format_stamp: "Formatted at 2026-08-06 16:12:09 on dist-test-slave-0b58"
server_key: "494d6823534bb62fb1ecb4cfa6036c52"
server_key_iv: "739502a722083f1b76816bc5a2bd4abd"
server_key_version: "encryptionkey@0"
I20260806 16:12:09.859318 20521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "a6d487d6b52949bf98fc4d6c1f19c98c"
format_stamp: "Formatted at 2026-08-06 16:12:09 on dist-test-slave-0b58"
server_key: "494d6823534bb62fb1ecb4cfa6036c52"
server_key_iv: "739502a722083f1b76816bc5a2bd4abd"
server_key_version: "encryptionkey@0"
I20260806 16:12:09.863276 20521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:09.865890 20536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:09.867197 20521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:09.867383 20521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "a6d487d6b52949bf98fc4d6c1f19c98c"
format_stamp: "Formatted at 2026-08-06 16:12:09 on dist-test-slave-0b58"
server_key: "494d6823534bb62fb1ecb4cfa6036c52"
server_key_iv: "739502a722083f1b76816bc5a2bd4abd"
server_key_version: "encryptionkey@0"
I20260806 16:12:09.867516 20521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:09.886696 20521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:09.889838 20521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:09.890070 20521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:09.898382 20521 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:33753
I20260806 16:12:09.898411 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:33753 every 8 connection(s)
I20260806 16:12:09.899443 20521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:09.900115 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 20521
I20260806 16:12:09.900240 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:09.900559 18792 external_mini_cluster.cc:1428] Setting key 6367420979619c059bc69ee58c294678
I20260806 16:12:09.902895 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:09.909940 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Bootstrap starting.
Aug 06 16:12:09 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:09.912587 20600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:09.913429 20600 log.cc:824] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:09.915498 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: No bootstrap required, opened a new log
I20260806 16:12:09.917326 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:09.902060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.18.90.62:33753)
0806 16:12:09.902530 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:09.902545 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:09.902583 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:12:09.903322 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:09.903340 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:09.903840 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:09.904238 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:09.905629 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:09.907228 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:09.908130 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:09.908732 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:09.908904 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:09.911756 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:09.911761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:09.911775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:09.911805 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:09.914318 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:09.914793 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:09.914799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:09.914812 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:09.914928 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:09.915226 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:09.915230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:09.915241 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:09.915698 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:09.915910 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:09.916220 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 16:12:09.916570 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":175,"threads_started":1}
I20260806 16:12:09.919054 20600 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33753 } }
I20260806 16:12:09.919281 20600 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:09.919344 20600 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d487d6b52949bf98fc4d6c1f19c98c, State: Initialized, Role: FOLLOWER
I20260806 16:12:09.919837 20600 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33753 } }
I20260806 16:12:09.920032 20600 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:09.920100 20600 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:09.920226 20600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:09.921208 20600 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33753 } }
I20260806 16:12:09.921564 20600 leader_election.cc:302] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6d487d6b52949bf98fc4d6c1f19c98c; no voters:
I20260806 16:12:09.921891 20600 leader_election.cc:288] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:09.922024 20605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:09.922251 20605 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [term 1 LEADER]: Becoming Leader. State: Replica: a6d487d6b52949bf98fc4d6c1f19c98c, State: Running, Role: LEADER
I20260806 16:12:09.922590 20605 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33753 } }
I20260806 16:12:09.923316 20600 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:09.924520 20606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33753 } } }
I20260806 16:12:09.924501 20607 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6d487d6b52949bf98fc4d6c1f19c98c. Latest consensus state: current_term: 1 leader_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d487d6b52949bf98fc4d6c1f19c98c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33753 } } }
I20260806 16:12:09.924692 20606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:09.924693 20607 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:09.925071 20614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:09.928364 20614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:09.934257 20614 catalog_manager.cc:1347] Generated new cluster ID: d5216d3e18c24ffbbd37d306b2249e7e
I20260806 16:12:09.934366 20614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:09.949716 20614 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:09.950758 20614 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:09.961390 20614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Generated new TSK 0
I20260806 16:12:09.962167 20614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:10.001720 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33753
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:10.147192 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:10.147486 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:10.147536 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:10.151575 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:10.151641 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:10.151681 20626 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 16:12:10.151695 20626 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:10.151727 20626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:10.157116 20626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:33753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:10.158211 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:10.159226 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:10.170058 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:10.170049 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:10.170177 20626 server_base.cc:1034] running on GCE node
W20260806 16:12:10.170049 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:10.170727 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:10.171474 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:10.172629 20626 hybrid_clock.cc:648] HybridClock initialized: now 1786032730172611 us; error 29 us; skew 500 ppm
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:10.175463 20626 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:10.176400 20626 webserver.cc:460] Webserver started at http://127.18.90.1:38813/ using document root <none> and password file <none>
I20260806 16:12:10.176959 20626 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:10.177017 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:10.177186 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:10.179275 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "b13e08bea60f453db445add611739a83"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "4eb9e8ca633c391a9cbea59691939d90"
server_key_iv: "8dc6afbad50c28bac9e99ae1a4b889fe"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.179885 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "b13e08bea60f453db445add611739a83"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "4eb9e8ca633c391a9cbea59691939d90"
server_key_iv: "8dc6afbad50c28bac9e99ae1a4b889fe"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.183794 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:10.186388 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.187685 20626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:10.187796 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "b13e08bea60f453db445add611739a83"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "4eb9e8ca633c391a9cbea59691939d90"
server_key_iv: "8dc6afbad50c28bac9e99ae1a4b889fe"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.187892 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:10.211620 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:10.215281 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:10.215509 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:10.217092 20626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:10.217144 20626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.217187 20626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:10.217247 20626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.228574 20626 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:42603
I20260806 16:12:10.228814 20758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:42603 every 8 connection(s)
I20260806 16:12:10.229823 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:10.230695 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 20626
I20260806 16:12:10.230850 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:10.231151 18792 external_mini_cluster.cc:1428] Setting key 6493c2e049161330b6948fbcbbb9b7ba
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:10.237457 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.218118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:51105 (local address 127.18.90.62:33753)
0806 16:12:10.218272 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.218276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.219573 (+ 1297us) server_negotiation.cc:407] Connection header received
0806 16:12:10.220843 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.220846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.220908 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.221005 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.222784 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.223743 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.225853 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.226243 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.226347 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.233338 (+ 6991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.233343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.233346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.233380 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.235008 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.235590 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.235593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.235594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.235675 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.236056 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.236062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.236067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.236365 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.236480 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.237041 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.237203 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:12:10.238188 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.218745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.18.90.1:51105)
0806 16:12:10.219404 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.219450 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.220573 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.221240 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.221253 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.221753 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.222568 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.222595 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.223967 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.223974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.225712 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.225723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.226612 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.226618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.226864 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.227644 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.227664 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.233146 (+ 5482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.235142 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.235148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.235165 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.235455 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.235783 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.235787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.235789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.235946 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.236488 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.236494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.236762 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.236931 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.237215 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":206,"threads_started":1}
I20260806 16:12:10.239691 20759 heartbeater.cc:347] Connected to a master server at 127.18.90.62:33753
I20260806 16:12:10.239876 20759 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:10.240535 20759 heartbeater.cc:511] Master 127.18.90.62:33753 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:10.242398 20552 ts_manager.cc:194] Registered new tserver with Master: b13e08bea60f453db445add611739a83 (127.18.90.1:42603)
I20260806 16:12:10.243801 20552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:51105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:12:10.250772 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.242076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:56711 (local address 127.18.90.62:33753)
0806 16:12:10.242191 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.242196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.242937 (+ 741us) server_negotiation.cc:407] Connection header received
0806 16:12:10.243010 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.243012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.243057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.243148 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.243708 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.244444 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.245645 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.246034 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.246173 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.247360 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.247366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.247371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.247403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.248980 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.249482 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.249485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.249486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.249560 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.249916 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.249919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.249920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.250106 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.250226 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.250537 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.250646 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:12:10.251056 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.242003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.18.90.1:56711)
0806 16:12:10.242844 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.242857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.242943 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.243177 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.243179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.243473 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.243594 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.243600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.244599 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.244602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.245506 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.245518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.246189 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.246193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.246290 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.246871 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.246883 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.247206 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.249095 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.249101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.249106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.249380 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.249651 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.249657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.249661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.249777 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.250230 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.250236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.250356 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.250491 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.250678 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:10.263046 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33753
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:10.380151 20769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:10.380522 20769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:10.380573 20769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:10.384635 20769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:10.384701 20769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:10.384743 20769 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 16:12:10.384758 20769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:10.384791 20769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:10.390213 20769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:33753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:10.391384 20769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:10.392604 20769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:10.402609 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:10.402618 20775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:10.402618 20777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:10.402715 20769 server_base.cc:1034] running on GCE node
I20260806 16:12:10.403350 20769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:10.403991 20769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:10.405164 20769 hybrid_clock.cc:648] HybridClock initialized: now 1786032730405126 us; error 44 us; skew 500 ppm
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:10.408170 20769 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:10.409181 20769 webserver.cc:460] Webserver started at http://127.18.90.2:38375/ using document root <none> and password file <none>
I20260806 16:12:10.409809 20769 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:10.409873 20769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:10.410128 20769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:10.412352 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "fa805dd8bd95434baf67283512004060"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "7976dd782c5b1770ad38f7dbb935a9d4"
server_key_iv: "05579252d5e2a77389499b4510ec6f9d"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.412971 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "fa805dd8bd95434baf67283512004060"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "7976dd782c5b1770ad38f7dbb935a9d4"
server_key_iv: "05579252d5e2a77389499b4510ec6f9d"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.416908 20769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:12:10.419592 20784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.421011 20769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:10.421147 20769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "fa805dd8bd95434baf67283512004060"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "7976dd782c5b1770ad38f7dbb935a9d4"
server_key_iv: "05579252d5e2a77389499b4510ec6f9d"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.421278 20769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:10.457360 20769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:10.460934 20769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:10.461148 20769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:10.462648 20769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:10.462729 20769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.462800 20769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:10.462841 20769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:10.474277 20769 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:40345
I20260806 16:12:10.475167 20769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:10.476058 20899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:40345 every 8 connection(s)
I20260806 16:12:10.477625 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.463651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:39671 (local address 127.18.90.62:33753)
0806 16:12:10.463785 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.463789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.464937 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 16:12:10.466052 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.466055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.466114 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.466245 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.467977 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.468811 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.470153 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.470530 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.470660 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.473618 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.473622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.473625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.473673 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.475340 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.475986 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.475990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.475992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.476083 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.476495 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.476499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.476501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.476703 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.476815 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.477320 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.477452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:10.478425 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.464208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.18.90.2:39671)
0806 16:12:10.464777 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.464825 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.465807 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.466485 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.466497 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.467019 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.467796 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.467819 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.468945 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.468949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.470022 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.470032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.470742 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.470749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.470942 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.471627 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.471649 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.473443 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.475471 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.475477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.475492 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.475859 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.476192 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.476195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.476197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.476394 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.476814 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.476821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.477074 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.477258 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.477591 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":147,"threads_started":1}
I20260806 16:12:10.479678 20900 heartbeater.cc:347] Connected to a master server at 127.18.90.62:33753
I20260806 16:12:10.479844 20900 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:10.480482 20900 heartbeater.cc:511] Master 127.18.90.62:33753 requested a full tablet report, sending...
I20260806 16:12:10.480818 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 20769
I20260806 16:12:10.480931 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:10.481190 18792 external_mini_cluster.cc:1428] Setting key 535cf75206713d5a8712ddf1931f83fe
I20260806 16:12:10.482544 20552 ts_manager.cc:194] Registered new tserver with Master: fa805dd8bd95434baf67283512004060 (127.18.90.2:40345)
I20260806 16:12:10.483294 20552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:39671
I20260806 16:12:10.491863 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.482246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:39279 (local address 127.18.90.62:33753)
0806 16:12:10.482526 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.482532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.483804 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 16:12:10.483917 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.483920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.483966 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.484048 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.484690 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.485423 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.486694 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.486898 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.486974 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.488384 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.488388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.488390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.488412 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.489848 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.490423 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.490426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.490427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.490509 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.490898 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.490902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.490904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.491237 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.491360 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.491633 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.491740 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260806 16:12:10.492002 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.482187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.18.90.2:39279)
0806 16:12:10.483704 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.483721 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.483830 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.484085 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.484091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.484386 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.484551 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.484563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.485570 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.485577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.486570 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.486585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.487015 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.487022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.487158 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.487682 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.487702 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.488200 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.489962 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.489968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.489972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.490285 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.490617 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.490622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.490626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.490741 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.491342 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.491345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.491463 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.491579 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.491769 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:10.511668 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33753
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:10.627704 20909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:10.628015 20909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:10.628067 20909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:10.632126 20909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:10.632202 20909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:10.632246 20909 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 16:12:10.632277 20909 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:10.632366 20909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:10.637737 20909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:33753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:10.638957 20909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:10.639969 20909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:10.650261 20914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:10.650363 20909 server_base.cc:1034] running on GCE node
W20260806 16:12:10.650305 20915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:10.650261 20917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:10.650985 20909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:10.651697 20909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:10.652894 20909 hybrid_clock.cc:648] HybridClock initialized: now 1786032730652853 us; error 52 us; skew 500 ppm
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:10.655889 20909 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:10.656963 20909 webserver.cc:460] Webserver started at http://127.18.90.3:46709/ using document root <none> and password file <none>
I20260806 16:12:10.657565 20909 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:10.657675 20909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:10.657917 20909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:10.660161 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "1235d1f352ea4522899cbb9466471b97"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "73380c18530909bc22214c68bd77dcde"
server_key_iv: "83e7a1fb6464b863fdd434cb04ee14a5"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.660836 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "1235d1f352ea4522899cbb9466471b97"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "73380c18530909bc22214c68bd77dcde"
server_key_iv: "83e7a1fb6464b863fdd434cb04ee14a5"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.664831 20909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:10.667459 20924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.668831 20909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:10.668990 20909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "1235d1f352ea4522899cbb9466471b97"
format_stamp: "Formatted at 2026-08-06 16:12:10 on dist-test-slave-0b58"
server_key: "73380c18530909bc22214c68bd77dcde"
server_key_iv: "83e7a1fb6464b863fdd434cb04ee14a5"
server_key_version: "encryptionkey@0"
I20260806 16:12:10.669121 20909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:10.695210 20909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:10.698688 20909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:10.698933 20909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:10.700531 20909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:10.700605 20909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:10.700675 20909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:10.700721 20909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:10.712098 20909 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:38743
I20260806 16:12:10.712169 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:38743 every 8 connection(s)
I20260806 16:12:10.713078 20909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:10.714712 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.701428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:49663 (local address 127.18.90.62:33753)
0806 16:12:10.701620 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.701627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.702547 (+ 920us) server_negotiation.cc:407] Connection header received
0806 16:12:10.703958 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.703965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.704039 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.704158 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.705444 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.705980 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.707189 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.707421 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.707533 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.710743 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.710747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.710749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.710775 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.712534 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.713063 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.713067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.713069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.713153 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.713592 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.713595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.713597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.713842 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.713944 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.714468 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.714581 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 16:12:10.715495 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.701836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.18.90.3:49663)
0806 16:12:10.702350 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.702414 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.703625 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.704367 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.704374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.704777 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.705278 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.705292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.706112 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.706116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.707046 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.707057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.707591 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.707596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.707774 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.708427 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.708443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.710557 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.712656 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.712662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.712677 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.712951 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.713265 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.713268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.713271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.713462 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.714003 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.714008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.714238 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.714398 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.714654 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260806 16:12:10.716773 21043 heartbeater.cc:347] Connected to a master server at 127.18.90.62:33753
I20260806 16:12:10.716913 21043 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:10.717491 21043 heartbeater.cc:511] Master 127.18.90.62:33753 requested a full tablet report, sending...
I20260806 16:12:10.718663 20553 ts_manager.cc:194] Registered new tserver with Master: 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743)
I20260806 16:12:10.719478 20553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:49663
I20260806 16:12:10.719540 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 20909
I20260806 16:12:10.719658 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:10.720007 18792 external_mini_cluster.cc:1428] Setting key 5912263279232396080b6642975df6f4
I20260806 16:12:10.722122 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:12:10.728297 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.719216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:60055 (local address 127.18.90.62:33753)
0806 16:12:10.719335 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.719341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.720610 (+ 1269us) server_negotiation.cc:407] Connection header received
0806 16:12:10.720728 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.720731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.720800 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.720885 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.721421 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.722115 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.723258 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.723488 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.723608 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.724998 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.725001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.725004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.725027 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.726593 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.727022 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.727026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.727027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.727112 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.727447 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.727451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.727453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.727633 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.727698 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.727991 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.728100 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:12:10.728314 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.719128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.18.90.3:60055)
0806 16:12:10.720518 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.720533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.720651 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.720924 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.720927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.721142 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.721258 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.721265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.722257 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.722261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.723137 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.723152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.723592 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.723599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.723737 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.724322 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.724341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.724864 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.726716 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.726719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.726721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.726919 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.727221 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.727224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.727226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.727312 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.727762 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.727764 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.727833 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.727937 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.728078 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"thread_start_us":117,"threads_started":1}
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:10.749487 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.741780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.18.90.62:33753)
0806 16:12:10.741913 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.741916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.741929 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:10.741965 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.741968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.742004 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.742070 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.742734 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.743233 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.744005 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.744208 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.744358 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.746653 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.746656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.746658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.746681 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.748092 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.748534 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.748537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.748538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.748612 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.748830 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.748832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.748833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.748983 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.749077 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.749255 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.749365 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:12:10.752485 20553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:10.755549 20553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:10.770108 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.764126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:40345 (local address 127.0.0.1:52376)
0806 16:12:10.765041 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.765098 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.765221 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.765842 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.765849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.765868 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:10.766021 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.766028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.767259 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.767264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.768552 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.768562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.769511 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.769515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.769628 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.769679 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.769804 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.769949 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":119,"threads_started":1}
I20260806 16:12:10.770135 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.764253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.18.90.2:40345)
0806 16:12:10.765279 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.765291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.765338 (+ 47us) server_negotiation.cc:407] Connection header received
0806 16:12:10.765424 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.765430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.765668 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.765849 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:10.766181 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.767148 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.768676 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.769377 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.769485 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.769655 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.769859 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.769978 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":109,"threads_started":1}
I20260806 16:12:10.770740 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.764662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.0.0.1:57012)
0806 16:12:10.765432 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.765445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.765526 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.766023 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.766028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.766045 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:10.766199 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.766207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.767697 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.767701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.769053 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.769065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.770209 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.770215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.770331 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.770425 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.770500 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.770576 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":193,"thread_start_us":79,"threads_started":1}
I20260806 16:12:10.770781 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.765040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.18.90.1:42603)
0806 16:12:10.765540 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.765549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.765574 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:12:10.765656 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.765661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.765876 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.766016 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:10.766346 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.767551 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.769161 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.769885 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.769984 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.770122 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.770538 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.770652 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":122,"threads_started":1}
I20260806 16:12:10.770795 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.764152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:38743 (local address 127.0.0.1:57822)
0806 16:12:10.765041 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.765513 (+ 472us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.765661 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.766200 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.766203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.766221 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:10.766336 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.766342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.767697 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.767701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.768958 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.768970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.770238 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.770242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.770442 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.770465 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.770540 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.770630 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":454,"thread_start_us":101,"threads_started":1}
I20260806 16:12:10.771245 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.764444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.18.90.3:38743)
0806 16:12:10.764906 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.764919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.765593 (+ 674us) server_negotiation.cc:407] Connection header received
0806 16:12:10.765818 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.765823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.766009 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.766179 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:10.766457 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.767551 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.769106 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.769897 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.769998 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.770153 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.771019 (+ 866us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.771112 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260806 16:12:10.773874 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 40bdad15d21941568f659bee1e555b0c (DEFAULT_TABLE table=test-table [id=a2e01e72bcdf41be96db037e98e02097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:10.773831 20975 tablet_service.cc:1467] Processing CreateTablet for tablet 40bdad15d21941568f659bee1e555b0c (DEFAULT_TABLE table=test-table [id=a2e01e72bcdf41be96db037e98e02097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:10.773831 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 40bdad15d21941568f659bee1e555b0c (DEFAULT_TABLE table=test-table [id=a2e01e72bcdf41be96db037e98e02097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:10.774889 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bdad15d21941568f659bee1e555b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:10.774917 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bdad15d21941568f659bee1e555b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:10.774914 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bdad15d21941568f659bee1e555b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:10.782137 21067 tablet_bootstrap.cc:492] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Bootstrap starting.
I20260806 16:12:10.782611 21068 tablet_bootstrap.cc:492] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Bootstrap starting.
I20260806 16:12:10.782999 21066 tablet_bootstrap.cc:492] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Bootstrap starting.
I20260806 16:12:10.785012 21066 tablet_bootstrap.cc:654] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:10.785106 21068 tablet_bootstrap.cc:654] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:10.785790 21066 log.cc:824] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:10.785941 21068 log.cc:824] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:10.786319 21067 tablet_bootstrap.cc:654] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:10.787132 21067 log.cc:824] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:10.788056 21068 tablet_bootstrap.cc:492] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: No bootstrap required, opened a new log
I20260806 16:12:10.788237 21068 ts_tablet_manager.cc:1397] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:12:10.788496 21066 tablet_bootstrap.cc:492] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: No bootstrap required, opened a new log
I20260806 16:12:10.788713 21066 ts_tablet_manager.cc:1397] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:12:10.789150 21067 tablet_bootstrap.cc:492] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: No bootstrap required, opened a new log
I20260806 16:12:10.789336 21067 ts_tablet_manager.cc:1397] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 16:12:10.792431 21066 raft_consensus.cc:348] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.792389 21068 raft_consensus.cc:348] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.792680 21066 raft_consensus.cc:374] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:10.792678 21068 raft_consensus.cc:374] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:10.792735 21066 raft_consensus.cc:729] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13e08bea60f453db445add611739a83, State: Initialized, Role: FOLLOWER
I20260806 16:12:10.792735 21068 raft_consensus.cc:729] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Initialized, Role: FOLLOWER
I20260806 16:12:10.793102 21067 raft_consensus.cc:348] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.793200 21066 consensus_queue.cc:260] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.793205 21068 consensus_queue.cc:260] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.793540 21067 raft_consensus.cc:374] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:10.793606 21067 raft_consensus.cc:729] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa805dd8bd95434baf67283512004060, State: Initialized, Role: FOLLOWER
I20260806 16:12:10.794016 21067 consensus_queue.cc:260] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.794221 21068 ts_tablet_manager.cc:1428] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 16:12:10.794224 21066 ts_tablet_manager.cc:1428] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:12:10.794574 20759 heartbeater.cc:503] Master 127.18.90.62:33753 was elected leader, sending a full tablet report...
I20260806 16:12:10.794682 21043 heartbeater.cc:503] Master 127.18.90.62:33753 was elected leader, sending a full tablet report...
I20260806 16:12:10.794848 21067 ts_tablet_manager.cc:1428] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:12:10.795131 20900 heartbeater.cc:503] Master 127.18.90.62:33753 was elected leader, sending a full tablet report...
I20260806 16:12:10.907689 21074 raft_consensus.cc:482] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:10.907966 21074 raft_consensus.cc:504] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.909626 21074 leader_election.cc:288] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743), b13e08bea60f453db445add611739a83 (127.18.90.1:42603)
I20260806 16:12:10.915241 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.909965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:59925 (local address 127.18.90.3:38743)
0806 16:12:10.910079 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.910085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.910312 (+ 227us) server_negotiation.cc:407] Connection header received
0806 16:12:10.910383 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.910388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.910456 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.910533 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:10.910984 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.911754 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.913982 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.914609 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.914730 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.914798 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.915012 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.915114 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:10.915262 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.909884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:38743 (local address 127.18.90.2:59925)
0806 16:12:10.910077 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.910094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.910203 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.910664 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.910670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.910691 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:10.910835 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.910844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.911876 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.911879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.913842 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.913854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.914743 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.914747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.914867 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.914896 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.914979 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.915062 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":97}
I20260806 16:12:10.915349 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.910770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:44833 (local address 127.18.90.1:42603)
0806 16:12:10.910918 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.910923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.910938 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:10.910989 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.910991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.911043 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.911125 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:10.911448 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.912097 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.913982 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.914630 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.914729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.914798 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.915138 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.915217 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:10.915751 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.910303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.18.90.2:44833)
0806 16:12:10.910683 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.910708 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.910816 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.911153 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.911156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.911172 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:10.911312 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.911318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.912294 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.912299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.913843 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.913854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.914863 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.914866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.914993 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.915020 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.915480 (+ 460us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.915574 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":105,"thread_start_us":140,"threads_started":1}
I20260806 16:12:10.916075 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bdad15d21941568f659bee1e555b0c" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1235d1f352ea4522899cbb9466471b97" is_pre_election: true
I20260806 16:12:10.916414 20993 raft_consensus.cc:2454] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 0.
I20260806 16:12:10.916381 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bdad15d21941568f659bee1e555b0c" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83" is_pre_election: true
I20260806 16:12:10.916725 20706 raft_consensus.cc:2454] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 0.
I20260806 16:12:10.916846 20788 leader_election.cc:302] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:10.917095 21074 raft_consensus.cc:2781] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:10.917178 21074 raft_consensus.cc:482] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:10.917241 21074 raft_consensus.cc:3037] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:10.918309 21074 raft_consensus.cc:504] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.918706 21074 leader_election.cc:288] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 election: Requested vote from peers 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743), b13e08bea60f453db445add611739a83 (127.18.90.1:42603)
I20260806 16:12:10.919052 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bdad15d21941568f659bee1e555b0c" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1235d1f352ea4522899cbb9466471b97"
I20260806 16:12:10.919165 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40bdad15d21941568f659bee1e555b0c" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83"
I20260806 16:12:10.919194 20993 raft_consensus.cc:3037] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:10.919302 20706 raft_consensus.cc:3037] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:10.920727 20706 raft_consensus.cc:2454] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 1.
I20260806 16:12:10.921041 20993 raft_consensus.cc:2454] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 1.
I20260806 16:12:10.921046 20785 leader_election.cc:302] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b13e08bea60f453db445add611739a83, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:10.921244 21074 raft_consensus.cc:2781] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:10.921509 21074 raft_consensus.cc:686] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 1 LEADER]: Becoming Leader. State: Replica: fa805dd8bd95434baf67283512004060, State: Running, Role: LEADER
I20260806 16:12:10.921865 21074 consensus_queue.cc:237] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } }
I20260806 16:12:10.925369 20553 catalog_manager.cc:5495] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa805dd8bd95434baf67283512004060 (127.18.90.2). New cstate: current_term: 1 leader_uuid: "fa805dd8bd95434baf67283512004060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } health_report { overall_health: UNKNOWN } } }
W20260806 16:12:10.966542 21044 tablet.cc:2369] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:12:10.976533 20901 tablet.cc:2369] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:12:10.981263 20760 tablet.cc:2369] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 16:12:10 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032729, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:10.986984 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.978258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33753 (local address 127.0.0.1:57292)
0806 16:12:10.978593 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:10.978605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:10.978753 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:10.979024 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:10.979026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:10.979475 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:10.979576 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.979583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.980418 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.980424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.981044 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:10.981054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.981562 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.981566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:10.981673 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.982309 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:10.982321 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:10.983816 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:10.985490 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.985500 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.985504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.985742 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.985998 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:10.986000 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:10.986001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:10.986077 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:10.986464 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:10.986470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:10.986584 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 16:12:10.986687 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 16:12:10.986822 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260806 16:12:10.987005 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:10.978309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.18.90.62:33753)
0806 16:12:10.978466 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:10.978473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:10.978686 (+ 213us) server_negotiation.cc:407] Connection header received
0806 16:12:10.978865 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:10.978868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:10.978913 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:10.978999 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:10.979724 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.980302 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.981182 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:10.981405 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:10.981523 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:10.983977 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:10.983980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:10.983982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:10.984004 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:10.985376 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.985833 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.985836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.985837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.985915 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:10.986172 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:10.986174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:10.986175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:10.986360 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:10.986439 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:10.986753 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 16:12:10.986881 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":50}
I20260806 16:12:10.991042 20553 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 16:12:10.991850 20553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:10.996809 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff1df32812a44598980020d9e1d5ff5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4d5d677526a42879393fe04ba25e600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:10.996809 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff1df32812a44598980020d9e1d5ff5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4d5d677526a42879393fe04ba25e600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:10.996997 20975 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff1df32812a44598980020d9e1d5ff5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4d5d677526a42879393fe04ba25e600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:10.997582 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff1df32812a44598980020d9e1d5ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:10.997582 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff1df32812a44598980020d9e1d5ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:10.997759 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff1df32812a44598980020d9e1d5ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:11.000566 21068 tablet_bootstrap.cc:492] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: Bootstrap starting.
I20260806 16:12:11.000712 21067 tablet_bootstrap.cc:492] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: Bootstrap starting.
I20260806 16:12:11.000903 21066 tablet_bootstrap.cc:492] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: Bootstrap starting.
I20260806 16:12:11.001507 21068 tablet_bootstrap.cc:654] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:11.001724 21067 tablet_bootstrap.cc:654] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:11.001869 21066 tablet_bootstrap.cc:654] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:11.002987 21068 tablet_bootstrap.cc:492] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: No bootstrap required, opened a new log
I20260806 16:12:11.003086 21068 ts_tablet_manager.cc:1397] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 16:12:11.003124 21066 tablet_bootstrap.cc:492] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: No bootstrap required, opened a new log
I20260806 16:12:11.003216 21066 ts_tablet_manager.cc:1397] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:11.003306 21067 tablet_bootstrap.cc:492] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: No bootstrap required, opened a new log
I20260806 16:12:11.003396 21067 ts_tablet_manager.cc:1397] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:11.003579 21068 raft_consensus.cc:348] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.003713 21068 raft_consensus.cc:374] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:11.003758 21068 raft_consensus.cc:729] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Initialized, Role: FOLLOWER
I20260806 16:12:11.003679 21066 raft_consensus.cc:348] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.003835 21066 raft_consensus.cc:374] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:11.003880 21066 raft_consensus.cc:729] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13e08bea60f453db445add611739a83, State: Initialized, Role: FOLLOWER
I20260806 16:12:11.003845 21067 raft_consensus.cc:348] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.003947 21067 raft_consensus.cc:374] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:11.003988 21067 raft_consensus.cc:729] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa805dd8bd95434baf67283512004060, State: Initialized, Role: FOLLOWER
I20260806 16:12:11.003930 21068 consensus_queue.cc:260] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.004017 21066 consensus_queue.cc:260] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.004128 21067 consensus_queue.cc:260] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.004314 21066 ts_tablet_manager.cc:1428] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:12:11.004314 21073 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.004424 21067 ts_tablet_manager.cc:1428] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:11.004462 21073 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.004417 21074 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.004549 21074 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.004570 21072 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.004719 21072 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.005506 21068 ts_tablet_manager.cc:1428] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 16:12:11.084940 21072 raft_consensus.cc:482] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:11.085104 21072 raft_consensus.cc:504] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.086407 21072 leader_election.cc:288] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa805dd8bd95434baf67283512004060 (127.18.90.2:40345), 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743)
I20260806 16:12:11.091497 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.086819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:57507 (local address 127.18.90.3:38743)
0806 16:12:11.086949 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.086955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.087346 (+ 391us) server_negotiation.cc:407] Connection header received
0806 16:12:11.087439 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.087442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.087498 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.087586 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:11.088546 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.089110 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.090375 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.090962 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.091043 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.091151 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.091325 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.091395 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:12:11.091482 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.086819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:56173 (local address 127.18.90.2:40345)
0806 16:12:11.086949 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.086954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.087271 (+ 317us) server_negotiation.cc:407] Connection header received
0806 16:12:11.087440 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.087443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.087500 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.087586 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:11.088028 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.088661 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.090125 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.090673 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.090784 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.090858 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.091241 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.091355 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:12:11.091516 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.086717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:40345 (local address 127.18.90.1:56173)
0806 16:12:11.087153 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.087170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.087313 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.087618 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.087621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.087662 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:11.087906 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.087914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.088776 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.088778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.089975 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.089986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.090914 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.090918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.091035 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.091065 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.091206 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.091299 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":65,"spinlock_wait_cycles":7296,"thread_start_us":149,"threads_started":1}
I20260806 16:12:11.091516 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.086717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:38743 (local address 127.18.90.1:57507)
0806 16:12:11.087254 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.087268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.087354 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.087618 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.087621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.087661 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:11.088419 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.088429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.089245 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.089249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.090267 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.090275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.091085 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.091090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.091206 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.091229 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.091298 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.091345 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":88,"spinlock_wait_cycles":30080,"thread_start_us":117,"threads_started":1}
I20260806 16:12:11.092407 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff1df32812a44598980020d9e1d5ff5" candidate_uuid: "b13e08bea60f453db445add611739a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1235d1f352ea4522899cbb9466471b97" is_pre_election: true
I20260806 16:12:11.092582 20993 raft_consensus.cc:2454] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b13e08bea60f453db445add611739a83 in term 0.
I20260806 16:12:11.092684 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff1df32812a44598980020d9e1d5ff5" candidate_uuid: "b13e08bea60f453db445add611739a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa805dd8bd95434baf67283512004060" is_pre_election: true
I20260806 16:12:11.092949 20854 raft_consensus.cc:2454] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b13e08bea60f453db445add611739a83 in term 0.
I20260806 16:12:11.092989 20645 leader_election.cc:302] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, b13e08bea60f453db445add611739a83; no voters:
I20260806 16:12:11.093336 21072 raft_consensus.cc:2781] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:11.093406 21072 raft_consensus.cc:482] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:11.093477 21072 raft_consensus.cc:3037] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:11.094780 21072 raft_consensus.cc:504] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.095197 21072 leader_election.cc:288] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [CANDIDATE]: Term 1 election: Requested vote from peers fa805dd8bd95434baf67283512004060 (127.18.90.2:40345), 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743)
I20260806 16:12:11.095597 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff1df32812a44598980020d9e1d5ff5" candidate_uuid: "b13e08bea60f453db445add611739a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1235d1f352ea4522899cbb9466471b97"
I20260806 16:12:11.095655 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff1df32812a44598980020d9e1d5ff5" candidate_uuid: "b13e08bea60f453db445add611739a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa805dd8bd95434baf67283512004060"
I20260806 16:12:11.095754 20993 raft_consensus.cc:3037] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:11.095763 20854 raft_consensus.cc:3037] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:11.096994 20993 raft_consensus.cc:2454] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b13e08bea60f453db445add611739a83 in term 1.
I20260806 16:12:11.096994 20854 raft_consensus.cc:2454] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b13e08bea60f453db445add611739a83 in term 1.
I20260806 16:12:11.097321 20644 leader_election.cc:302] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b13e08bea60f453db445add611739a83, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:11.097539 21072 raft_consensus.cc:2781] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:11.097813 21072 raft_consensus.cc:686] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [term 1 LEADER]: Becoming Leader. State: Replica: b13e08bea60f453db445add611739a83, State: Running, Role: LEADER
I20260806 16:12:11.098169 21072 consensus_queue.cc:237] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.100440 21089 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: TxnStatusTablet state changed. Reason: New leader b13e08bea60f453db445add611739a83. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.100553 21089 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:11.101140 21091 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 16:12:11.101372 21091 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 16:12:11.101816 20552 catalog_manager.cc:5495] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 reported cstate change: term changed from 0 to 1, leader changed from <none> to b13e08bea60f453db445add611739a83 (127.18.90.1). New cstate: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } health_report { overall_health: UNKNOWN } } }
I20260806 16:12:11.129590 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.125310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.0.0.1:57016)
0806 16:12:11.125456 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.125469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.125556 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.125893 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.125895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.125905 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:11.126065 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.126071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.127139 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.127142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.127756 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.127764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.128367 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.128370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.128442 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.128496 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.129330 (+ 834us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.129342 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.129392 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.129471 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 16:12:11.129611 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.125457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.18.90.1:42603)
0806 16:12:11.125586 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.125591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.125607 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:12:11.125693 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.125696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.125761 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.125873 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.126193 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.127032 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.127869 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.128216 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.128366 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.128648 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.129214 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.129349 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.129407 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.129471 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:12:11.134920 20993 raft_consensus.cc:1261] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Refusing update from remote peer b13e08bea60f453db445add611739a83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:11.134971 20854 raft_consensus.cc:1261] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Refusing update from remote peer b13e08bea60f453db445add611739a83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:11.135664 21089 consensus_queue.cc:1035] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:11.135867 21072 consensus_queue.cc:1035] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:11.139227 21074 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: TxnStatusTablet state changed. Reason: New leader b13e08bea60f453db445add611739a83. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.139387 21074 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.139894 21073 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: TxnStatusTablet state changed. Reason: New leader b13e08bea60f453db445add611739a83. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.140008 21073 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.142581 21094 mvcc.cc:205] Tried to move back new op lower bound from 7315590066721370112 to 7315590066580832256. Current Snapshot: MvccSnapshot[applied={T|T < 7315590066721370112}]
I20260806 16:12:11.142833 21073 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.142957 21073 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P 1235d1f352ea4522899cbb9466471b97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.143383 21089 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.143510 21089 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:11.144443 21095 mvcc.cc:205] Tried to move back new op lower bound from 7315590066721370112 to 7315590066580832256. Current Snapshot: MvccSnapshot[applied={T|T < 7315590066721370112}]
I20260806 16:12:11.144701 21072 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.144825 21072 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P b13e08bea60f453db445add611739a83: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:11.145550 21074 tablet_replica.cc:439] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b13e08bea60f453db445add611739a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } } }
I20260806 16:12:11.145653 21074 tablet_replica.cc:442] T 0ff1df32812a44598980020d9e1d5ff5 P fa805dd8bd95434baf67283512004060: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:11.148524 21096 mvcc.cc:205] Tried to move back new op lower bound from 7315590066721370112 to 7315590066580832256. Current Snapshot: MvccSnapshot[applied={T|T < 7315590066721370112}]
I20260806 16:12:11.158478 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.154429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.18.90.2:40345)
0806 16:12:11.154580 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.154586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.154601 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:11.154679 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.154681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.154744 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.154854 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.155111 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.155749 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.156791 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.157147 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.157257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.157506 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.158075 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.158190 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.158270 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.158330 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:12:11.170965 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.167118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:52393 (local address 127.18.90.1:42603)
0806 16:12:11.167259 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.167263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.167382 (+ 119us) server_negotiation.cc:407] Connection header received
0806 16:12:11.167545 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.167550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.167605 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.167695 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.168015 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.168601 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.169852 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.170068 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.170151 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.170523 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.170596 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.170650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.170802 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.170854 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:11.171031 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.166960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.18.90.2:52393)
0806 16:12:11.167284 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.167300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.167431 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.167714 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.167717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.167724 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:11.167861 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.167866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.168710 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.168713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.169717 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.169728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.170201 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.170205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.170310 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.170364 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.170678 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.170684 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.170770 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.170853 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
I20260806 16:12:11.178813 20993 raft_consensus.cc:1261] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Refusing update from remote peer fa805dd8bd95434baf67283512004060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:11.179039 20706 raft_consensus.cc:1261] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Refusing update from remote peer fa805dd8bd95434baf67283512004060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:11.179287 21074 consensus_queue.cc:1035] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:11.179575 21076 consensus_queue.cc:1035] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:11.205633 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.201303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:42775 (local address 127.18.90.2:40345)
0806 16:12:11.201430 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.201435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.201906 (+ 471us) server_negotiation.cc:407] Connection header received
0806 16:12:11.202024 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.202027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.202087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.202165 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.202365 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.203117 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.204347 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.204668 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.204784 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.205160 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.205256 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.205339 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.205431 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.205485 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:12:11.205633 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.201187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:40345 (local address 127.18.90.1:42775)
0806 16:12:11.201791 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.201809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.201923 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.202168 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.202169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.202175 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:11.202270 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.202274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.203234 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.203236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.204063 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.204074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.204873 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.204877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.204971 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.205028 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.205348 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.205358 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.205421 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.205504 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":130,"threads_started":1}
I20260806 16:12:11.235994 20551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:11.236203 20551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:11.239343 20551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Sending DeleteTablet for 3 replicas of tablet 40bdad15d21941568f659bee1e555b0c
I20260806 16:12:11.240175 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 40bdad15d21941568f659bee1e555b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:11 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:52376
I20260806 16:12:11.240638 20975 tablet_service.cc:1514] Processing DeleteTablet for tablet 40bdad15d21941568f659bee1e555b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:11 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260806 16:12:11.241577 20691 tablet_service.cc:1514] Processing DeleteTablet for tablet 40bdad15d21941568f659bee1e555b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:11 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57012
I20260806 16:12:11.242935 20551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:11.243407 20551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:11.249194 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0f156535124ceaacf96f158082b103 (DEFAULT_TABLE table=test-table [id=21cf32361ddc4056846b795b799bcb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:11.249318 20975 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0f156535124ceaacf96f158082b103 (DEFAULT_TABLE table=test-table [id=21cf32361ddc4056846b795b799bcb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:11.249507 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0f156535124ceaacf96f158082b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:11.249616 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0f156535124ceaacf96f158082b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:11.251953 21067 tablet_bootstrap.cc:492] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Bootstrap starting.
I20260806 16:12:11.252188 21068 tablet_bootstrap.cc:492] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Bootstrap starting.
I20260806 16:12:11.252381 21116 tablet_replica.cc:330] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: stopping tablet replica
I20260806 16:12:11.252749 21116 raft_consensus.cc:2229] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:11.252959 21067 tablet_bootstrap.cc:654] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:11.253151 21068 tablet_bootstrap.cc:654] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:11.253155 21116 raft_consensus.cc:2258] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:11.254565 21116 ts_tablet_manager.cc:1905] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:11.255086 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0f156535124ceaacf96f158082b103 (DEFAULT_TABLE table=test-table [id=21cf32361ddc4056846b795b799bcb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:11.255399 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0f156535124ceaacf96f158082b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:11.258127 21066 tablet_bootstrap.cc:492] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Bootstrap starting.
I20260806 16:12:11.259109 21066 tablet_bootstrap.cc:654] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:11.260578 21068 tablet_bootstrap.cc:492] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: No bootstrap required, opened a new log
I20260806 16:12:11.260671 21068 ts_tablet_manager.cc:1397] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20260806 16:12:11.261155 21068 raft_consensus.cc:348] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.261283 21068 raft_consensus.cc:374] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:11.261318 21068 raft_consensus.cc:729] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Initialized, Role: FOLLOWER
I20260806 16:12:11.261440 21068 consensus_queue.cc:260] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.261690 21068 ts_tablet_manager.cc:1428] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:12:11.261929 21116 ts_tablet_manager.cc:1918] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:11.262029 21116 log.cc:1196] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/40bdad15d21941568f659bee1e555b0c
I20260806 16:12:11.262349 21116 ts_tablet_manager.cc:1939] T 40bdad15d21941568f659bee1e555b0c P fa805dd8bd95434baf67283512004060: Deleting consensus metadata
I20260806 16:12:11.263585 20539 catalog_manager.cc:4841] TS fa805dd8bd95434baf67283512004060 (127.18.90.2:40345): tablet 40bdad15d21941568f659bee1e555b0c (table test-table [id=a2e01e72bcdf41be96db037e98e02097]) successfully deleted
I20260806 16:12:11.264544 21067 tablet_bootstrap.cc:492] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: No bootstrap required, opened a new log
I20260806 16:12:11.264628 21067 ts_tablet_manager.cc:1397] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260806 16:12:11.265055 21067 raft_consensus.cc:348] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.265172 21067 raft_consensus.cc:374] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:11.265210 21067 raft_consensus.cc:729] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa805dd8bd95434baf67283512004060, State: Initialized, Role: FOLLOWER
I20260806 16:12:11.265349 21067 consensus_queue.cc:260] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.265591 21067 ts_tablet_manager.cc:1428] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:11.265254 21117 tablet_replica.cc:330] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: stopping tablet replica
I20260806 16:12:11.266547 21117 raft_consensus.cc:2229] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:11.266808 21117 raft_consensus.cc:2258] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:11.267043 21118 tablet_replica.cc:330] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: stopping tablet replica
I20260806 16:12:11.267522 21118 raft_consensus.cc:2229] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:11.267817 21118 raft_consensus.cc:2258] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:11.268085 21066 tablet_bootstrap.cc:492] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: No bootstrap required, opened a new log
I20260806 16:12:11.268409 21066 ts_tablet_manager.cc:1397] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260806 16:12:11.268908 21066 raft_consensus.cc:348] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.269033 21066 raft_consensus.cc:374] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:11.269083 21066 raft_consensus.cc:729] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13e08bea60f453db445add611739a83, State: Initialized, Role: FOLLOWER
I20260806 16:12:11.269207 21066 consensus_queue.cc:260] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.269477 21066 ts_tablet_manager.cc:1428] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:11.269703 21117 ts_tablet_manager.cc:1905] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:11.271622 21118 ts_tablet_manager.cc:1905] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:11.272945 21117 ts_tablet_manager.cc:1918] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:11.273036 21117 log.cc:1196] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/40bdad15d21941568f659bee1e555b0c
I20260806 16:12:11.273463 21117 ts_tablet_manager.cc:1939] T 40bdad15d21941568f659bee1e555b0c P 1235d1f352ea4522899cbb9466471b97: Deleting consensus metadata
I20260806 16:12:11.274539 21118 ts_tablet_manager.cc:1918] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:11.274664 21118 log.cc:1196] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/40bdad15d21941568f659bee1e555b0c
I20260806 16:12:11.274817 20540 catalog_manager.cc:4841] TS 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743): tablet 40bdad15d21941568f659bee1e555b0c (table test-table [id=a2e01e72bcdf41be96db037e98e02097]) successfully deleted
I20260806 16:12:11.275007 21118 ts_tablet_manager.cc:1939] T 40bdad15d21941568f659bee1e555b0c P b13e08bea60f453db445add611739a83: Deleting consensus metadata
I20260806 16:12:11.276149 20537 catalog_manager.cc:4841] TS b13e08bea60f453db445add611739a83 (127.18.90.1:42603): tablet 40bdad15d21941568f659bee1e555b0c (table test-table [id=a2e01e72bcdf41be96db037e98e02097]) successfully deleted
I20260806 16:12:11.472736 21073 raft_consensus.cc:482] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:11.472888 21073 raft_consensus.cc:504] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.474092 21073 leader_election.cc:288] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b13e08bea60f453db445add611739a83 (127.18.90.1:42603), fa805dd8bd95434baf67283512004060 (127.18.90.2:40345)
I20260806 16:12:11.479239 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.474395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.18.90.3:54823)
0806 16:12:11.474650 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.474668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.474828 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.475518 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.475520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.475533 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:11.475629 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.475634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.476365 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.476369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.477860 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.477872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.478760 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.478764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.478876 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.478904 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.478987 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.479061 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":101,"thread_start_us":80,"threads_started":1}
I20260806 16:12:11.479252 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.475011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:54823 (local address 127.18.90.1:42603)
0806 16:12:11.475186 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.475193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.475208 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:11.475273 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.475275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.475312 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.475409 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:11.475734 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.476213 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.478017 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.478616 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.478721 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.478787 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.479022 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.479126 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 16:12:11.479337 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.474922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:42323 (local address 127.18.90.2:40345)
0806 16:12:11.475081 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.475086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.475105 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:12:11.475224 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.475227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.475275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.475358 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:11.475664 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.476455 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.478017 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.478616 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.478705 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.478773 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.479145 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.479217 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:12:11.479717 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f156535124ceaacf96f158082b103" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83" is_pre_election: true
I20260806 16:12:11.479900 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.474442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:40345 (local address 127.18.90.3:42323)
0806 16:12:11.475029 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.475044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.475130 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.475400 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.475403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.475419 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:11.475547 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.475554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.476563 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.476566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.477863 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.477872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.478877 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.478880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.479006 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.479030 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.479677 (+ 647us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.479760 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":42,"thread_start_us":81,"threads_started":1}
I20260806 16:12:11.479887 20706 raft_consensus.cc:2454] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 0.
I20260806 16:12:11.480244 20925 leader_election.cc:302] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, b13e08bea60f453db445add611739a83; no voters:
I20260806 16:12:11.480492 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f156535124ceaacf96f158082b103" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa805dd8bd95434baf67283512004060" is_pre_election: true
I20260806 16:12:11.480551 21073 raft_consensus.cc:2781] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:11.480623 21073 raft_consensus.cc:482] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:11.480647 20854 raft_consensus.cc:2454] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 0.
I20260806 16:12:11.480677 21073 raft_consensus.cc:3037] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:11.481830 21073 raft_consensus.cc:504] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.482203 21073 leader_election.cc:288] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 election: Requested vote from peers b13e08bea60f453db445add611739a83 (127.18.90.1:42603), fa805dd8bd95434baf67283512004060 (127.18.90.2:40345)
I20260806 16:12:11.482563 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f156535124ceaacf96f158082b103" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83"
I20260806 16:12:11.482556 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0f156535124ceaacf96f158082b103" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa805dd8bd95434baf67283512004060"
I20260806 16:12:11.482677 20706 raft_consensus.cc:3037] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:11.482689 20854 raft_consensus.cc:3037] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:11.483798 20706 raft_consensus.cc:2454] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 1.
I20260806 16:12:11.483794 20854 raft_consensus.cc:2454] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 1.
I20260806 16:12:11.484629 20927 leader_election.cc:302] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:11.484815 21073 raft_consensus.cc:2781] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:11.484992 21073 raft_consensus.cc:686] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 1 LEADER]: Becoming Leader. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Running, Role: LEADER
I20260806 16:12:11.485234 21073 consensus_queue.cc:237] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:11.487720 20551 catalog_manager.cc:5495] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1235d1f352ea4522899cbb9466471b97 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "1235d1f352ea4522899cbb9466471b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:11.511538 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.508381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.18.90.3:38743)
0806 16:12:11.508518 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.508522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.508567 (+ 45us) server_negotiation.cc:407] Connection header received
0806 16:12:11.508713 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.508715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.508763 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.508862 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.509036 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.509521 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.510271 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.510480 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.510556 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.510779 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.511221 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.511306 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.511381 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.511426 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:12:11.521857 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.518655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:44801 (local address 127.18.90.1:42603)
0806 16:12:11.518789 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.518792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.518877 (+ 85us) server_negotiation.cc:407] Connection header received
0806 16:12:11.518972 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.518975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.519018 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.519086 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.519337 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.519841 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.520745 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.520959 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.521041 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.521368 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.521447 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.521516 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.521684 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.521734 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:12:11.521898 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.518518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.18.90.3:44801)
0806 16:12:11.518795 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.518809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.518901 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.519107 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.519109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.519116 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:11.519232 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.519236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.519933 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.519936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.520627 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.520637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.521074 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.521077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.521152 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.521198 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.521572 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.521579 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.521661 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.521757 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20260806 16:12:11.528331 20854 raft_consensus.cc:1261] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Refusing update from remote peer 1235d1f352ea4522899cbb9466471b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:11.528354 20706 raft_consensus.cc:1261] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Refusing update from remote peer 1235d1f352ea4522899cbb9466471b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:11.528821 21124 consensus_queue.cc:1035] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:11.529008 21073 consensus_queue.cc:1035] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:11.546800 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.543348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:38743 (local address 127.18.90.1:40593)
0806 16:12:11.543471 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:11.543487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:11.543563 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:11.543877 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:11.543881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:11.543888 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:11.543989 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.543995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.544855 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.544859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.545445 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:11.545497 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.546075 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.546078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:11.546160 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.546188 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.546510 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.546516 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:12:11.546601 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:12:11.546687 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 16:12:11.546800 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:11.543542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:40593 (local address 127.18.90.3:38743)
0806 16:12:11.543679 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:11.543684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:11.543695 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:12:11.543741 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:11.543743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:11.543778 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:11.543860 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:11.544098 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.544735 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.545610 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:11.545937 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:11.546068 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:11.546308 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:11.546391 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:11.546484 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:11.546617 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 16:12:11.546668 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 16:12:11.547523 21095 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 16:12:13.107493 20551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:13.107678 20551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:13.110093 20551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Sending DeleteTablet for 3 replicas of tablet 3a0f156535124ceaacf96f158082b103
I20260806 16:12:13.110713 20691 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f156535124ceaacf96f158082b103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:13 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57012
I20260806 16:12:13.110704 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f156535124ceaacf96f158082b103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:13 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:52376
I20260806 16:12:13.110760 20975 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0f156535124ceaacf96f158082b103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:13 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260806 16:12:13.111079 21158 tablet_replica.cc:330] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: stopping tablet replica
I20260806 16:12:13.111104 21160 tablet_replica.cc:330] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: stopping tablet replica
I20260806 16:12:13.111228 21158 raft_consensus.cc:2229] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:13.111240 21160 raft_consensus.cc:2229] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:13.111349 21159 tablet_replica.cc:330] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: stopping tablet replica
I20260806 16:12:13.111359 21158 raft_consensus.cc:2258] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:13.111588 21160 raft_consensus.cc:2258] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:13.111831 21159 raft_consensus.cc:2229] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:13.112008 21159 raft_consensus.cc:2258] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:13.112054 21158 ts_tablet_manager.cc:1905] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:13.112337 21160 ts_tablet_manager.cc:1905] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:13.115353 21160 ts_tablet_manager.cc:1918] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:12:13.115368 21159 ts_tablet_manager.cc:1905] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:13.115458 21160 log.cc:1196] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/3a0f156535124ceaacf96f158082b103
I20260806 16:12:13.115477 21158 ts_tablet_manager.cc:1918] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:12:13.115543 21158 log.cc:1196] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/3a0f156535124ceaacf96f158082b103
I20260806 16:12:13.115795 21160 ts_tablet_manager.cc:1939] T 3a0f156535124ceaacf96f158082b103 P 1235d1f352ea4522899cbb9466471b97: Deleting consensus metadata
I20260806 16:12:13.115849 21158 ts_tablet_manager.cc:1939] T 3a0f156535124ceaacf96f158082b103 P b13e08bea60f453db445add611739a83: Deleting consensus metadata
I20260806 16:12:13.116667 20540 catalog_manager.cc:4841] TS 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743): tablet 3a0f156535124ceaacf96f158082b103 (table test-table [id=21cf32361ddc4056846b795b799bcb21]) successfully deleted
I20260806 16:12:13.116909 20537 catalog_manager.cc:4841] TS b13e08bea60f453db445add611739a83 (127.18.90.1:42603): tablet 3a0f156535124ceaacf96f158082b103 (table test-table [id=21cf32361ddc4056846b795b799bcb21]) successfully deleted
I20260806 16:12:13.119544 21159 ts_tablet_manager.cc:1918] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:12:13.119639 21159 log.cc:1196] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/3a0f156535124ceaacf96f158082b103
I20260806 16:12:13.119920 21159 ts_tablet_manager.cc:1939] T 3a0f156535124ceaacf96f158082b103 P fa805dd8bd95434baf67283512004060: Deleting consensus metadata
I20260806 16:12:13.120893 20539 catalog_manager.cc:4841] TS fa805dd8bd95434baf67283512004060 (127.18.90.2:40345): tablet 3a0f156535124ceaacf96f158082b103 (table test-table [id=21cf32361ddc4056846b795b799bcb21]) successfully deleted
Aug 06 16:12:13 dist-test-slave-0b58 krb5kdc[20508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:12:13 dist-test-slave-0b58 krb5kdc[20508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:13.153059 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:13.133561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.18.90.62:33753)
0806 16:12:13.133925 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:13.133933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:13.133950 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:12:13.134030 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:13.134032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:13.134074 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:13.134161 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:13.134856 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:13.135413 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:13.136226 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:13.136521 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:13.136606 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:13.149648 (+ 13042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:13.149652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:13.149654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:13.149691 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:13.151346 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:13.151806 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:13.151809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:13.151810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:13.151895 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:13.152145 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:13.152147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:13.152148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:13.152414 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:13.152528 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:13.152730 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 16:12:13.152841 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260806 16:12:13.155864 20551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:13.156391 20551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:13.161243 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 7d55c47a209145279ebd6fef5bd98d44 (DEFAULT_TABLE table=test-table [id=cd04769b2daf4cca8bcfd97f6d61b3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:13.161230 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 7d55c47a209145279ebd6fef5bd98d44 (DEFAULT_TABLE table=test-table [id=cd04769b2daf4cca8bcfd97f6d61b3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:13.161343 20975 tablet_service.cc:1467] Processing CreateTablet for tablet 7d55c47a209145279ebd6fef5bd98d44 (DEFAULT_TABLE table=test-table [id=cd04769b2daf4cca8bcfd97f6d61b3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:13.161579 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d55c47a209145279ebd6fef5bd98d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:13.161580 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d55c47a209145279ebd6fef5bd98d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:13.161595 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d55c47a209145279ebd6fef5bd98d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:13.164132 21170 tablet_bootstrap.cc:492] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Bootstrap starting.
I20260806 16:12:13.164505 21169 tablet_bootstrap.cc:492] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Bootstrap starting.
I20260806 16:12:13.164736 21171 tablet_bootstrap.cc:492] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Bootstrap starting.
I20260806 16:12:13.165135 21170 tablet_bootstrap.cc:654] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:13.165405 21169 tablet_bootstrap.cc:654] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:13.165578 21171 tablet_bootstrap.cc:654] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:13.166631 21170 tablet_bootstrap.cc:492] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: No bootstrap required, opened a new log
I20260806 16:12:13.166667 21169 tablet_bootstrap.cc:492] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: No bootstrap required, opened a new log
I20260806 16:12:13.166749 21170 ts_tablet_manager.cc:1397] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:13.166774 21169 ts_tablet_manager.cc:1397] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:13.167192 21169 raft_consensus.cc:348] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.167212 21170 raft_consensus.cc:348] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.167322 21169 raft_consensus.cc:374] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:13.167344 21170 raft_consensus.cc:374] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:13.167369 21169 raft_consensus.cc:729] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa805dd8bd95434baf67283512004060, State: Initialized, Role: FOLLOWER
I20260806 16:12:13.167388 21170 raft_consensus.cc:729] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Initialized, Role: FOLLOWER
I20260806 16:12:13.167512 21169 consensus_queue.cc:260] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.167519 21170 consensus_queue.cc:260] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.167826 21170 ts_tablet_manager.cc:1428] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:13.167886 21169 ts_tablet_manager.cc:1428] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:13.168642 21171 tablet_bootstrap.cc:492] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: No bootstrap required, opened a new log
I20260806 16:12:13.168746 21171 ts_tablet_manager.cc:1397] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 16:12:13.169212 21171 raft_consensus.cc:348] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.169354 21171 raft_consensus.cc:374] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:13.169407 21171 raft_consensus.cc:729] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13e08bea60f453db445add611739a83, State: Initialized, Role: FOLLOWER
I20260806 16:12:13.169548 21171 consensus_queue.cc:260] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.169849 21171 ts_tablet_manager.cc:1428] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:13.226971 21124 raft_consensus.cc:482] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:13.227125 21124 raft_consensus.cc:504] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.227583 21124 leader_election.cc:288] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b13e08bea60f453db445add611739a83 (127.18.90.1:42603), fa805dd8bd95434baf67283512004060 (127.18.90.2:40345)
I20260806 16:12:13.228024 20853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d55c47a209145279ebd6fef5bd98d44" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa805dd8bd95434baf67283512004060" is_pre_election: true
I20260806 16:12:13.228022 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d55c47a209145279ebd6fef5bd98d44" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83" is_pre_election: true
I20260806 16:12:13.228217 20706 raft_consensus.cc:2454] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 0.
I20260806 16:12:13.228230 20853 raft_consensus.cc:2454] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 0.
I20260806 16:12:13.228602 20927 leader_election.cc:302] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:13.228801 21124 raft_consensus.cc:2781] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:13.228863 21124 raft_consensus.cc:482] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:13.228904 21124 raft_consensus.cc:3037] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:13.229941 21124 raft_consensus.cc:504] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.230316 21124 leader_election.cc:288] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 election: Requested vote from peers b13e08bea60f453db445add611739a83 (127.18.90.1:42603), fa805dd8bd95434baf67283512004060 (127.18.90.2:40345)
I20260806 16:12:13.230631 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d55c47a209145279ebd6fef5bd98d44" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83"
I20260806 16:12:13.230649 20853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d55c47a209145279ebd6fef5bd98d44" candidate_uuid: "1235d1f352ea4522899cbb9466471b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa805dd8bd95434baf67283512004060"
I20260806 16:12:13.230752 20706 raft_consensus.cc:3037] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:13.230778 20853 raft_consensus.cc:3037] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:13.231853 20853 raft_consensus.cc:2454] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 1.
I20260806 16:12:13.231926 20706 raft_consensus.cc:2454] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1235d1f352ea4522899cbb9466471b97 in term 1.
I20260806 16:12:13.232228 20927 leader_election.cc:302] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:13.232416 21124 raft_consensus.cc:2781] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:13.232520 21124 raft_consensus.cc:686] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 1 LEADER]: Becoming Leader. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Running, Role: LEADER
I20260806 16:12:13.232681 21124 consensus_queue.cc:237] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.237435 20552 catalog_manager.cc:5495] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1235d1f352ea4522899cbb9466471b97 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "1235d1f352ea4522899cbb9466471b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:13.251447 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:13.248588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.18.90.3:38743)
0806 16:12:13.248851 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:13.248858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:13.248870 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:13.248909 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:13.248911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:13.248950 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:13.249024 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:13.249235 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:13.249746 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:13.250567 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:13.250789 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:13.250878 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:13.251086 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:13.251153 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:13.251207 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:13.251287 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 16:12:13.251332 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":117,"threads_started":1}
I20260806 16:12:13.257309 20853 raft_consensus.cc:1261] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Refusing update from remote peer 1235d1f352ea4522899cbb9466471b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:13.257491 20706 raft_consensus.cc:1261] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Refusing update from remote peer 1235d1f352ea4522899cbb9466471b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:13.257862 21124 consensus_queue.cc:1035] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:13.258037 21152 consensus_queue.cc:1035] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:13.289606 20552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:13.289768 20552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:13.291728 20552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Sending DeleteTablet for 3 replicas of tablet 7d55c47a209145279ebd6fef5bd98d44
I20260806 16:12:13.292408 20691 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d55c47a209145279ebd6fef5bd98d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:13 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57012
I20260806 16:12:13.292517 20975 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d55c47a209145279ebd6fef5bd98d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:13 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260806 16:12:13.292523 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d55c47a209145279ebd6fef5bd98d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:13 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:52376
I20260806 16:12:13.292611 21158 tablet_replica.cc:330] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: stopping tablet replica
I20260806 16:12:13.292721 21159 tablet_replica.cc:330] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: stopping tablet replica
I20260806 16:12:13.292732 21160 tablet_replica.cc:330] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: stopping tablet replica
I20260806 16:12:13.292737 21158 raft_consensus.cc:2229] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:13.292851 21158 raft_consensus.cc:2258] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:13.292837 21159 raft_consensus.cc:2229] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:13.292855 21160 raft_consensus.cc:2229] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:13.292951 21159 raft_consensus.cc:2258] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:13.293056 21160 raft_consensus.cc:2258] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:13.293040 20552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:13.293488 20552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:13.293759 21159 ts_tablet_manager.cc:1905] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:13.293882 21158 ts_tablet_manager.cc:1905] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:13.293937 21160 ts_tablet_manager.cc:1905] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:13.296445 21160 ts_tablet_manager.cc:1918] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:13.296445 21159 ts_tablet_manager.cc:1918] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:13.296519 21159 log.cc:1196] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/7d55c47a209145279ebd6fef5bd98d44
I20260806 16:12:13.296519 21160 log.cc:1196] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/7d55c47a209145279ebd6fef5bd98d44
I20260806 16:12:13.296576 21158 ts_tablet_manager.cc:1918] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:13.296651 21158 log.cc:1196] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/7d55c47a209145279ebd6fef5bd98d44
I20260806 16:12:13.296803 21160 ts_tablet_manager.cc:1939] T 7d55c47a209145279ebd6fef5bd98d44 P 1235d1f352ea4522899cbb9466471b97: Deleting consensus metadata
I20260806 16:12:13.296804 21159 ts_tablet_manager.cc:1939] T 7d55c47a209145279ebd6fef5bd98d44 P fa805dd8bd95434baf67283512004060: Deleting consensus metadata
I20260806 16:12:13.296932 21158 ts_tablet_manager.cc:1939] T 7d55c47a209145279ebd6fef5bd98d44 P b13e08bea60f453db445add611739a83: Deleting consensus metadata
I20260806 16:12:13.297525 20540 catalog_manager.cc:4841] TS 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743): tablet 7d55c47a209145279ebd6fef5bd98d44 (table test-table [id=cd04769b2daf4cca8bcfd97f6d61b3de]) successfully deleted
I20260806 16:12:13.297623 20537 catalog_manager.cc:4841] TS b13e08bea60f453db445add611739a83 (127.18.90.1:42603): tablet 7d55c47a209145279ebd6fef5bd98d44 (table test-table [id=cd04769b2daf4cca8bcfd97f6d61b3de]) successfully deleted
I20260806 16:12:13.298717 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 (DEFAULT_TABLE table=test-table [id=ebd4a75ff90341e7a3e9a44774768458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:13.298722 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 (DEFAULT_TABLE table=test-table [id=ebd4a75ff90341e7a3e9a44774768458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:13.298734 20975 tablet_service.cc:1467] Processing CreateTablet for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 (DEFAULT_TABLE table=test-table [id=ebd4a75ff90341e7a3e9a44774768458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:13.298965 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:13.298965 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:13.298972 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:13.300757 21169 tablet_bootstrap.cc:492] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Bootstrap starting.
I20260806 16:12:13.301126 21171 tablet_bootstrap.cc:492] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83: Bootstrap starting.
I20260806 16:12:13.301435 20539 catalog_manager.cc:4841] TS fa805dd8bd95434baf67283512004060 (127.18.90.2:40345): tablet 7d55c47a209145279ebd6fef5bd98d44 (table test-table [id=cd04769b2daf4cca8bcfd97f6d61b3de]) successfully deleted
I20260806 16:12:13.301640 21169 tablet_bootstrap.cc:654] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:13.301990 21171 tablet_bootstrap.cc:654] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:13.302544 21170 tablet_bootstrap.cc:492] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Bootstrap starting.
I20260806 16:12:13.302780 21169 tablet_bootstrap.cc:492] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: No bootstrap required, opened a new log
I20260806 16:12:13.302881 21169 ts_tablet_manager.cc:1397] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:13.303268 21169 raft_consensus.cc:348] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.303400 21170 tablet_bootstrap.cc:654] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:13.303417 21169 raft_consensus.cc:374] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:13.303452 21169 raft_consensus.cc:729] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa805dd8bd95434baf67283512004060, State: Initialized, Role: FOLLOWER
I20260806 16:12:13.303589 21169 consensus_queue.cc:260] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.303933 21169 ts_tablet_manager.cc:1428] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:13.304289 21171 tablet_bootstrap.cc:492] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83: No bootstrap required, opened a new log
I20260806 16:12:13.304407 21171 ts_tablet_manager.cc:1397] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:13.304816 21171 raft_consensus.cc:348] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.304934 21171 raft_consensus.cc:374] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:13.304968 21171 raft_consensus.cc:729] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13e08bea60f453db445add611739a83, State: Initialized, Role: FOLLOWER
I20260806 16:12:13.305092 21171 consensus_queue.cc:260] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.305418 21171 ts_tablet_manager.cc:1428] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:13.305879 21170 tablet_bootstrap.cc:492] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: No bootstrap required, opened a new log
I20260806 16:12:13.305953 21170 ts_tablet_manager.cc:1397] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 16:12:13.306453 21170 raft_consensus.cc:348] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.306577 21170 raft_consensus.cc:374] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:13.306613 21170 raft_consensus.cc:729] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1235d1f352ea4522899cbb9466471b97, State: Initialized, Role: FOLLOWER
I20260806 16:12:13.306748 21170 consensus_queue.cc:260] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.307049 21170 ts_tablet_manager.cc:1428] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:13.402247 21175 raft_consensus.cc:482] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:13.402428 21175 raft_consensus.cc:504] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.402920 21175 leader_election.cc:288] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b13e08bea60f453db445add611739a83 (127.18.90.1:42603), 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743)
I20260806 16:12:13.403515 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ae9c2f9a4f3fb89f01ecfedd4a90" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83" is_pre_election: true
I20260806 16:12:13.403517 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ae9c2f9a4f3fb89f01ecfedd4a90" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1235d1f352ea4522899cbb9466471b97" is_pre_election: true
I20260806 16:12:13.403677 20993 raft_consensus.cc:2454] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 0.
I20260806 16:12:13.403688 20706 raft_consensus.cc:2454] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 0.
I20260806 16:12:13.404008 20788 leader_election.cc:302] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1235d1f352ea4522899cbb9466471b97, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:13.404220 21175 raft_consensus.cc:2781] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:13.404309 21175 raft_consensus.cc:482] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:13.404368 21175 raft_consensus.cc:3037] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:13.405411 21175 raft_consensus.cc:504] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.405819 21175 leader_election.cc:288] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 election: Requested vote from peers b13e08bea60f453db445add611739a83 (127.18.90.1:42603), 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743)
I20260806 16:12:13.406128 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ae9c2f9a4f3fb89f01ecfedd4a90" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13e08bea60f453db445add611739a83"
I20260806 16:12:13.406188 20993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ae9c2f9a4f3fb89f01ecfedd4a90" candidate_uuid: "fa805dd8bd95434baf67283512004060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1235d1f352ea4522899cbb9466471b97"
I20260806 16:12:13.406306 20993 raft_consensus.cc:3037] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:13.406297 20706 raft_consensus.cc:3037] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:13.407298 20993 raft_consensus.cc:2454] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 1.
I20260806 16:12:13.407305 20706 raft_consensus.cc:2454] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa805dd8bd95434baf67283512004060 in term 1.
I20260806 16:12:13.407626 20785 leader_election.cc:302] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b13e08bea60f453db445add611739a83, fa805dd8bd95434baf67283512004060; no voters:
I20260806 16:12:13.407833 21175 raft_consensus.cc:2781] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:13.408051 21175 raft_consensus.cc:686] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 1 LEADER]: Becoming Leader. State: Replica: fa805dd8bd95434baf67283512004060, State: Running, Role: LEADER
I20260806 16:12:13.408206 21175 consensus_queue.cc:237] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } }
I20260806 16:12:13.409933 20552 catalog_manager.cc:5495] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa805dd8bd95434baf67283512004060 (127.18.90.2). New cstate: current_term: 1 leader_uuid: "fa805dd8bd95434baf67283512004060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa805dd8bd95434baf67283512004060" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 40345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 } health_report { overall_health: UNKNOWN } } }
I20260806 16:12:13.442962 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:13.439661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.18.90.2:40345)
0806 16:12:13.439993 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:13.440001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:13.440012 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:12:13.440050 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:13.440052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:13.440090 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:13.440184 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:13.440507 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:13.441032 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:13.441874 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:13.442108 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:13.442221 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:13.442489 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:13.442564 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:13.442649 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:13.442748 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 16:12:13.442800 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":137,"threads_started":1}
I20260806 16:12:13.448941 20706 raft_consensus.cc:1261] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P b13e08bea60f453db445add611739a83 [term 1 FOLLOWER]: Refusing update from remote peer fa805dd8bd95434baf67283512004060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:13.448941 20993 raft_consensus.cc:1261] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Refusing update from remote peer fa805dd8bd95434baf67283512004060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:13.449357 21185 consensus_queue.cc:1035] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13e08bea60f453db445add611739a83" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:13.449523 21175 consensus_queue.cc:1035] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1235d1f352ea4522899cbb9466471b97" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 16:12:13.462968 21139 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 16:12:15.015863 20551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:15.016010 20551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:15.018545 20551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6d487d6b52949bf98fc4d6c1f19c98c: Sending DeleteTablet for 3 replicas of tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90
I20260806 16:12:15.019037 20691 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:15 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57012
I20260806 16:12:15.019233 20975 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:15 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260806 16:12:15.019233 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:15 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:52376
I20260806 16:12:15.019522 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 20626
I20260806 16:12:15.020572 21215 tablet_replica.cc:330] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: stopping tablet replica
I20260806 16:12:15.020728 21215 raft_consensus.cc:2229] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:15.020852 21215 raft_consensus.cc:2258] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:15.021036 21214 tablet_replica.cc:330] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: stopping tablet replica
I20260806 16:12:15.021198 21214 raft_consensus.cc:2229] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:15.021382 21214 raft_consensus.cc:2258] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:15.021978 21214 ts_tablet_manager.cc:1905] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:15.024500 21214 ts_tablet_manager.cc:1918] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:12:15.024605 21214 log.cc:1196] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/27d5ae9c2f9a4f3fb89f01ecfedd4a90
I20260806 16:12:15.024726 21215 ts_tablet_manager.cc:1905] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:15.024894 21214 ts_tablet_manager.cc:1939] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P fa805dd8bd95434baf67283512004060: Deleting consensus metadata
I20260806 16:12:15.025702 20539 catalog_manager.cc:4841] TS fa805dd8bd95434baf67283512004060 (127.18.90.2:40345): tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 (table test-table [id=ebd4a75ff90341e7a3e9a44774768458]) successfully deleted
I20260806 16:12:15.027175 21215 ts_tablet_manager.cc:1918] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:12:15.027268 21215 log.cc:1196] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/27d5ae9c2f9a4f3fb89f01ecfedd4a90
I20260806 16:12:15.027539 21215 ts_tablet_manager.cc:1939] T 27d5ae9c2f9a4f3fb89f01ecfedd4a90 P 1235d1f352ea4522899cbb9466471b97: Deleting consensus metadata
I20260806 16:12:15.028368 20540 catalog_manager.cc:4841] TS 1235d1f352ea4522899cbb9466471b97 (127.18.90.3:38743): tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90 (table test-table [id=ebd4a75ff90341e7a3e9a44774768458]) successfully deleted
I20260806 16:12:15.031458 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 20769
W20260806 16:12:15.031473 20537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:42603 (error 108)
I20260806 16:12:15.033444 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:15.032926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:42603 (local address 127.0.0.1:57028)
0806 16:12:15.033289 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:15.033358 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:42603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":83,"threads_started":1}
W20260806 16:12:15.033674 20537 catalog_manager.cc:4584] TS b13e08bea60f453db445add611739a83 (127.18.90.1:42603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27d5ae9c2f9a4f3fb89f01ecfedd4a90: Network error: Client connection negotiation failed: client connection to 127.18.90.1:42603: connect: Connection refused (error 111)
I20260806 16:12:15.039743 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 20909
I20260806 16:12:15.047348 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 20521
2026-08-06T16:12:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6516 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:15 dist-test-slave-0b58 krb5kdc[21220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:15 dist-test-slave-0b58 krb5kdc[21220](info): set up 2 sockets
Aug 06 16:12:15 dist-test-slave-0b58 krb5kdc[21220](info): commencing operation
krb5kdc: starting...
W20260806 16:12:16.101545 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:16.231418 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38221
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:38221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:16.347889 21233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:16.348165 21233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:16.348222 21233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:16.352322 21233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:16.352393 21233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:16.352412 21233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:16.352429 21233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:16.352445 21233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:16.358011 21233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:38221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:16.359229 21233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:16.360369 21233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:16.369251 21241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:16.369328 21238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:16.369258 21239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:16.369306 21233 server_base.cc:1034] running on GCE node
I20260806 16:12:16.370107 21233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:16.371290 21233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:16.372453 21233 hybrid_clock.cc:648] HybridClock initialized: now 1786032736372417 us; error 49 us; skew 500 ppm
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:16.375121 21233 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:16.376153 21233 webserver.cc:460] Webserver started at http://127.18.90.62:41311/ using document root <none> and password file <none>
I20260806 16:12:16.376761 21233 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:16.376837 21233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:16.377066 21233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:16.379237 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "a5a3edf8e9fa476b96f6d80f016cffd4"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "e0e67e486fc7d0923b5b565507f67d50"
server_key_iv: "6f62ee1f5840ba2224da714529c0252c"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.379823 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "a5a3edf8e9fa476b96f6d80f016cffd4"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "e0e67e486fc7d0923b5b565507f67d50"
server_key_iv: "6f62ee1f5840ba2224da714529c0252c"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.383694 21233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:16.386298 21248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.387513 21233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:16.387660 21233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "a5a3edf8e9fa476b96f6d80f016cffd4"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "e0e67e486fc7d0923b5b565507f67d50"
server_key_iv: "6f62ee1f5840ba2224da714529c0252c"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.387786 21233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:16.402132 21233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:16.405175 21233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:16.405391 21233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:16.412928 21233 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:38221
I20260806 16:12:16.412954 21301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:38221 every 8 connection(s)
I20260806 16:12:16.413910 21233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:16.416740 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:16.418928 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21233
I20260806 16:12:16.419023 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:16.419319 18792 external_mini_cluster.cc:1428] Setting key cacc546245edfab811717c7f2ddc577a
I20260806 16:12:16.423451 21302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4: Bootstrap starting.
I20260806 16:12:16.426157 21302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:16.427094 21302 log.cc:824] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:16.429162 21302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4: No bootstrap required, opened a new log
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:16.432967 21302 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38221 } }
I20260806 16:12:16.433269 21302 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:16.433353 21302 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a3edf8e9fa476b96f6d80f016cffd4, State: Initialized, Role: FOLLOWER
I20260806 16:12:16.433866 21302 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38221 } }
I20260806 16:12:16.434043 21302 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:16.434119 21302 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:16.434221 21302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:16.435407 21302 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38221 } }
I20260806 16:12:16.435757 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:16.420818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.18.90.62:38221)
0806 16:12:16.421260 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:16.421282 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:16.421328 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:12:16.422292 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:16.422317 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:16.422745 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:16.423104 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:16.423899 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.425400 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.426253 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.426968 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.427136 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:16.429932 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:16.429937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:16.429950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:16.429977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:16.432405 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:16.432899 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:16.432903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:16.432908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:16.433018 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:16.433377 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:16.433381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:16.433382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:16.433894 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:16.434099 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:16.434435 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 16:12:16.434737 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":107,"threads_started":1}
I20260806 16:12:16.435760 21302 leader_election.cc:302] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5a3edf8e9fa476b96f6d80f016cffd4; no voters:
I20260806 16:12:16.436173 21302 leader_election.cc:288] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:16.436328 21307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:16.436568 21307 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [term 1 LEADER]: Becoming Leader. State: Replica: a5a3edf8e9fa476b96f6d80f016cffd4, State: Running, Role: LEADER
I20260806 16:12:16.437402 21302 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:16.437666 21307 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38221 } }
I20260806 16:12:16.439644 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38221 } } }
I20260806 16:12:16.439990 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:16.439777 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a3edf8e9fa476b96f6d80f016cffd4. Latest consensus state: current_term: 1 leader_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a3edf8e9fa476b96f6d80f016cffd4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38221 } } }
I20260806 16:12:16.440366 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4 [sys.catalog]: This master's current role is: LEADER
W20260806 16:12:16.442663 21321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:12:16.442751 21321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:12:16.442836 21322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:16.443545 21322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:16.449373 21322 catalog_manager.cc:1347] Generated new cluster ID: a8fdbb33110e4d33960f83213849f51c
I20260806 16:12:16.449441 21322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:16.484845 21322 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:16.486240 21322 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:16.493247 21322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5a3edf8e9fa476b96f6d80f016cffd4: Generated new TSK 0
I20260806 16:12:16.494083 21322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:16.531002 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38221
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:16.647965 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:16.648347 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:16.648442 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:16.652585 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:16.652693 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:16.652794 21328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:16.658221 21328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:16.659389 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:16.660537 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:16.670578 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:16.670571 21333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:16.670578 21336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:16.670957 21328 server_base.cc:1034] running on GCE node
I20260806 16:12:16.671424 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:16.672075 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:16.673266 21328 hybrid_clock.cc:648] HybridClock initialized: now 1786032736673230 us; error 56 us; skew 500 ppm
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:16.676128 21328 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:16.677100 21328 webserver.cc:460] Webserver started at http://127.18.90.1:40043/ using document root <none> and password file <none>
I20260806 16:12:16.677685 21328 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:16.677778 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:16.678031 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:16.680105 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "4b681da11c2d4388ab4d09c6a73801f8"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "5cf323182e8d203e85a096799bd87dbd"
server_key_iv: "c21bb3958d79a78779509e1b17441a3c"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.680756 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "4b681da11c2d4388ab4d09c6a73801f8"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "5cf323182e8d203e85a096799bd87dbd"
server_key_iv: "c21bb3958d79a78779509e1b17441a3c"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.684500 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:12:16.687001 21343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.688354 21328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:16.688474 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "4b681da11c2d4388ab4d09c6a73801f8"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "5cf323182e8d203e85a096799bd87dbd"
server_key_iv: "c21bb3958d79a78779509e1b17441a3c"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.688596 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:16.720950 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:16.724617 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:16.724846 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:16.725505 21328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:16.726397 21328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:16.726467 21328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.726538 21328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:16.726581 21328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.735656 21328 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:40729
I20260806 16:12:16.735688 21458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:40729 every 8 connection(s)
I20260806 16:12:16.736615 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:16.738991 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21328
I20260806 16:12:16.739126 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:16.739527 18792 external_mini_cluster.cc:1428] Setting key 76d9093204a70a14af8abc53b1f25797
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:16.753248 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:16.738567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:55787 (local address 127.18.90.62:38221)
0806 16:12:16.738767 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:16.738772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:16.739799 (+ 1027us) server_negotiation.cc:407] Connection header received
0806 16:12:16.741375 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:16.741379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:16.741432 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:16.741524 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:16.742718 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.743510 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.745105 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.745587 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.745730 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:16.748471 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:16.748477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:16.748483 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:16.748524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:16.750902 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:16.751451 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:16.751456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:16.751458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:16.751561 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:16.751925 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:16.751932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:16.751934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:16.752201 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:16.752341 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:16.752876 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 16:12:16.753031 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:12:16.754135 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:16.739067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38221 (local address 127.18.90.1:55787)
0806 16:12:16.739608 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:16.739662 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:16.741167 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:16.741682 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:16.741689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:16.742049 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:16.742557 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:16.742572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.743645 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.743652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:16.744970 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:16.744981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.745777 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.745780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:16.745952 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:16.746666 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:16.746682 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:16.748230 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:16.751032 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:16.751037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:16.751052 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:16.751318 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:16.751657 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:16.751659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:16.751661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:16.751806 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:16.752347 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:16.752352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:16.752609 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 16:12:16.752765 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 16:12:16.753085 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260806 16:12:16.755529 21459 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38221
I20260806 16:12:16.755944 21459 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:12:16.756611 21459 heartbeater.cc:511] Master 127.18.90.62:38221 requested a full tablet report, sending...
I20260806 16:12:16.758211 21266 ts_manager.cc:194] Registered new tserver with Master: 4b681da11c2d4388ab4d09c6a73801f8 (127.18.90.1:40729)
I20260806 16:12:16.759850 21266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:55787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:16.773763 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38221
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:16.884305 21464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:16.884595 21464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:16.884676 21464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:16.888895 21464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:16.888957 21464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:16.889058 21464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:16.894670 21464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:16.895788 21464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:16.896859 21464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:16.906414 21472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:16.906378 21470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:16.906378 21469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:16.906757 21464 server_base.cc:1034] running on GCE node
I20260806 16:12:16.907192 21464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:16.907837 21464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:16.909044 21464 hybrid_clock.cc:648] HybridClock initialized: now 1786032736909014 us; error 40 us; skew 500 ppm
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:16.911821 21464 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:16.912716 21464 webserver.cc:460] Webserver started at http://127.18.90.2:39399/ using document root <none> and password file <none>
I20260806 16:12:16.913275 21464 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:16.913336 21464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:16.913511 21464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:16.915575 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "e45ad7bb9a9d4fcf86ad6bd595dce641"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "25333bdf18807ee26d5becd1fd2d5271"
server_key_iv: "8786fed155bd6d7b0b0d9d3bb144c831"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.916092 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "e45ad7bb9a9d4fcf86ad6bd595dce641"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "25333bdf18807ee26d5becd1fd2d5271"
server_key_iv: "8786fed155bd6d7b0b0d9d3bb144c831"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.919884 21464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 16:12:16.922479 21479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.923969 21464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:12:16.924062 21464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "e45ad7bb9a9d4fcf86ad6bd595dce641"
format_stamp: "Formatted at 2026-08-06 16:12:16 on dist-test-slave-0b58"
server_key: "25333bdf18807ee26d5becd1fd2d5271"
server_key_iv: "8786fed155bd6d7b0b0d9d3bb144c831"
server_key_version: "encryptionkey@0"
I20260806 16:12:16.924180 21464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:16.945573 21464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:16.949116 21464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:16.949280 21464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:16.949853 21464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:16.950680 21464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:16.950727 21464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.950768 21464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:16.950793 21464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:16.959535 21464 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38801
I20260806 16:12:16.959559 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38801 every 8 connection(s)
I20260806 16:12:16.960618 21464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:16.970188 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21464
I20260806 16:12:16.970310 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
Aug 06 16:12:16 dist-test-slave-0b58 krb5kdc[21220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032736, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:16.970579 18792 external_mini_cluster.cc:1428] Setting key 0f1911f532aa54c84771c6fbd707785b
I20260806 16:12:16.975232 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:16.962423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:56499 (local address 127.18.90.62:38221)
0806 16:12:16.962564 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:16.962567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:16.963365 (+ 798us) server_negotiation.cc:407] Connection header received
0806 16:12:16.964793 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:16.964797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:16.964861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:16.964967 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:16.966210 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.966745 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.967937 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.968158 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.968304 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:16.971003 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:16.971015 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:16.971018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:16.971043 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:16.972700 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:16.973494 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:16.973497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:16.973499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:16.973586 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:16.973993 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:16.974000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:16.974003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:16.974239 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:16.974332 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:16.975002 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 16:12:16.975106 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:12:16.976552 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:16.962748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38221 (local address 127.18.90.2:56499)
0806 16:12:16.963196 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:16.963243 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:16.964559 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:16.965151 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:16.965157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:16.965533 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:16.966033 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:16.966047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.966880 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.966883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:16.967797 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:16.967807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:16.968371 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:16.968375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:16.968546 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:16.969126 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:16.969142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:16.970798 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:16.973039 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:16.973046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:16.973062 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:16.973372 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:16.973701 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:16.973704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:16.973706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:16.973858 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:16.974359 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:16.974368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:16.974732 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 16:12:16.974966 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 16:12:16.975371 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260806 16:12:16.977923 21598 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38221
I20260806 16:12:16.978191 21598 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:16.978704 21598 heartbeater.cc:511] Master 127.18.90.62:38221 requested a full tablet report, sending...
I20260806 16:12:16.979808 21266 ts_manager.cc:194] Registered new tserver with Master: e45ad7bb9a9d4fcf86ad6bd595dce641 (127.18.90.2:38801)
I20260806 16:12:16.980458 21266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:56499
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:16.999253 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38221
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:17.113173 21603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:17.113471 21603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:17.113548 21603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:17.117528 21603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:17.117615 21603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:17.117707 21603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:17.122886 21603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:17.123960 21603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:17.125013 21603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:17.135005 21608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:17.135152 21611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:17.134994 21609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:17.135089 21603 server_base.cc:1034] running on GCE node
I20260806 16:12:17.135720 21603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:17.136459 21603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:17.137645 21603 hybrid_clock.cc:648] HybridClock initialized: now 1786032737137620 us; error 43 us; skew 500 ppm
Aug 06 16:12:17 dist-test-slave-0b58 krb5kdc[21220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032737, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:17.140707 21603 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:17.141570 21603 webserver.cc:460] Webserver started at http://127.18.90.3:36249/ using document root <none> and password file <none>
I20260806 16:12:17.142103 21603 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:17.142164 21603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:17.142339 21603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:17.144502 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "be31a65fc3be45d58f1f87889c3dd47c"
format_stamp: "Formatted at 2026-08-06 16:12:17 on dist-test-slave-0b58"
server_key: "80d6f2483e22cdd386679b6450c45f36"
server_key_iv: "4ff998fa54127786478bb4db499fc3ad"
server_key_version: "encryptionkey@0"
I20260806 16:12:17.145042 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "be31a65fc3be45d58f1f87889c3dd47c"
format_stamp: "Formatted at 2026-08-06 16:12:17 on dist-test-slave-0b58"
server_key: "80d6f2483e22cdd386679b6450c45f36"
server_key_iv: "4ff998fa54127786478bb4db499fc3ad"
server_key_version: "encryptionkey@0"
I20260806 16:12:17.148989 21603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:17.151504 21618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:17.152817 21603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:12:17.152943 21603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "be31a65fc3be45d58f1f87889c3dd47c"
format_stamp: "Formatted at 2026-08-06 16:12:17 on dist-test-slave-0b58"
server_key: "80d6f2483e22cdd386679b6450c45f36"
server_key_iv: "4ff998fa54127786478bb4db499fc3ad"
server_key_version: "encryptionkey@0"
I20260806 16:12:17.153054 21603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:17.163669 21603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:17.167071 21603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:17.167299 21603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:17.167951 21603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:17.168916 21603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:17.168965 21603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:17.169034 21603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:17.169070 21603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:17.178634 21603 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:32783
I20260806 16:12:17.178656 21734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:32783 every 8 connection(s)
I20260806 16:12:17.179636 21603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:17.186079 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21603
I20260806 16:12:17.186228 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:17.186599 18792 external_mini_cluster.cc:1428] Setting key aafcd8621408e7f9ac4db14e7aee751c
Aug 06 16:12:17 dist-test-slave-0b58 krb5kdc[21220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032737, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:17.195094 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:17.181680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:41641 (local address 127.18.90.62:38221)
0806 16:12:17.181831 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:17.181835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:17.182681 (+ 846us) server_negotiation.cc:407] Connection header received
0806 16:12:17.184056 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:17.184060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:17.184133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:17.184224 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:17.185512 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:17.186302 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:17.188117 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:17.188397 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:17.188522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:17.191220 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:17.191224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:17.191226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:17.191262 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:17.192940 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:17.193480 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:17.193483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:17.193485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:17.193576 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:17.193923 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:17.193928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:17.193930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:17.194142 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:17.194254 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:17.194780 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 16:12:17.194917 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:17.195962 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:17.182045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38221 (local address 127.18.90.3:41641)
0806 16:12:17.182507 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:17.182554 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:17.183819 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:17.184439 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:17.184447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:17.184842 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:17.185352 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:17.185365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:17.186469 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:17.186472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:17.187983 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:17.187994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:17.188569 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:17.188574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:17.188773 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:17.189482 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:17.189500 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:17.191049 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:17.193063 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:17.193069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:17.193086 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:17.193373 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:17.193669 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:17.193673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:17.193675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:17.193826 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:17.194256 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:17.194264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:17.194532 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 16:12:17.194735 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 16:12:17.195088 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260806 16:12:17.197165 21735 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38221
I20260806 16:12:17.197431 21735 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:17.197918 21735 heartbeater.cc:511] Master 127.18.90.62:38221 requested a full tablet report, sending...
I20260806 16:12:17.198951 21266 ts_manager.cc:194] Registered new tserver with Master: be31a65fc3be45d58f1f87889c3dd47c (127.18.90.3:32783)
I20260806 16:12:17.199589 21266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:41641
I20260806 16:12:17.200933 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 16:12:17.218943 21305 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:17.217248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.18.90.62:38221)
0806 16:12:17.217436 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:17.217441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:17.217606 (+ 165us) server_negotiation.cc:407] Connection header received
0806 16:12:17.217813 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:17.217816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:17.217864 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:17.217961 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:17.218772 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42328: BlockingRecv error: recv got EOF from 127.0.0.1:42328 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:12:17.220328 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21328
I20260806 16:12:17.230219 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21464
I20260806 16:12:17.239646 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21603
I20260806 16:12:17.251843 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21233
2026-08-06T16:12:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2204 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:17 dist-test-slave-0b58 krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:17 dist-test-slave-0b58 krb5kdc[21747](info): set up 2 sockets
Aug 06 16:12:17 dist-test-slave-0b58 krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260806 16:12:18.313846 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:18 dist-test-slave-0b58 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:18.443981 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45861
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:18.558542 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:18.558823 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:18.558863 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:18.562916 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:18.562979 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:18.562999 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:18.563014 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:18.563030 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:18.568374 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:45861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:18.569442 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:18.570437 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:18.579051 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:18.579054 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:18.579056 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:18.579108 21760 server_base.cc:1034] running on GCE node
I20260806 16:12:18.579967 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:18.581173 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:18.582334 21760 hybrid_clock.cc:648] HybridClock initialized: now 1786032738582306 us; error 42 us; skew 500 ppm
Aug 06 16:12:18 dist-test-slave-0b58 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:18.589782 21760 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:18.590811 21760 webserver.cc:460] Webserver started at http://127.18.90.62:40607/ using document root <none> and password file <none>
I20260806 16:12:18.591410 21760 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:18.591504 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:18.591744 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:18.593950 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "be4f58994aa8445f9d02e38ebc9a06d6"
format_stamp: "Formatted at 2026-08-06 16:12:18 on dist-test-slave-0b58"
server_key: "a65b7211469e8786f989686c3f45c60d"
server_key_iv: "c1a3fb51ec0a9103713da6caefdd749e"
server_key_version: "encryptionkey@0"
I20260806 16:12:18.594547 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "be4f58994aa8445f9d02e38ebc9a06d6"
format_stamp: "Formatted at 2026-08-06 16:12:18 on dist-test-slave-0b58"
server_key: "a65b7211469e8786f989686c3f45c60d"
server_key_iv: "c1a3fb51ec0a9103713da6caefdd749e"
server_key_version: "encryptionkey@0"
I20260806 16:12:18.598487 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:18.601249 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:18.602592 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:18.602758 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "be4f58994aa8445f9d02e38ebc9a06d6"
format_stamp: "Formatted at 2026-08-06 16:12:18 on dist-test-slave-0b58"
server_key: "a65b7211469e8786f989686c3f45c60d"
server_key_iv: "c1a3fb51ec0a9103713da6caefdd749e"
server_key_version: "encryptionkey@0"
I20260806 16:12:18.602908 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:18.614194 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:18.617543 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:18.617749 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:18.625000 21760 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:45861
I20260806 16:12:18.625319 21831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:45861 every 8 connection(s)
I20260806 16:12:18.626003 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:18.628886 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:18.630801 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21760
I20260806 16:12:18.630919 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:18.631225 18792 external_mini_cluster.cc:1428] Setting key 8c71583b6cb4adacd3a34246156fec27
I20260806 16:12:18.635712 21832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6: Bootstrap starting.
I20260806 16:12:18.638469 21832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:18.639449 21832 log.cc:824] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:12:18 dist-test-slave-0b58 krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:18.641635 21832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6: No bootstrap required, opened a new log
I20260806 16:12:18.646687 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:18.632694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.18.90.62:45861)
0806 16:12:18.633131 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:18.633147 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:18.633184 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:12:18.633823 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:18.633841 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:18.634168 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:18.634449 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:18.635423 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:18.636507 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:18.637731 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:18.638212 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:18.638327 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:18.641271 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:18.641276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:18.641291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:18.641316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:18.643289 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:18.643735 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:18.643742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:18.643748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:18.643914 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:18.644195 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:18.644198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:18.644202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:18.644810 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:18.645004 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:18.645258 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 16:12:18.645579 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
I20260806 16:12:18.646610 21832 raft_consensus.cc:348] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45861 } }
I20260806 16:12:18.646888 21832 raft_consensus.cc:374] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:18.646955 21832 raft_consensus.cc:729] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4f58994aa8445f9d02e38ebc9a06d6, State: Initialized, Role: FOLLOWER
I20260806 16:12:18.647442 21832 consensus_queue.cc:260] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45861 } }
I20260806 16:12:18.647604 21832 raft_consensus.cc:388] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:18.647675 21832 raft_consensus.cc:482] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:18.647779 21832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:18.648985 21832 raft_consensus.cc:504] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45861 } }
I20260806 16:12:18.649385 21832 leader_election.cc:302] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be4f58994aa8445f9d02e38ebc9a06d6; no voters:
I20260806 16:12:18.649749 21832 leader_election.cc:288] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:18.649796 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:12:18.649829 21837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:18.650038 21837 raft_consensus.cc:686] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [term 1 LEADER]: Becoming Leader. State: Replica: be4f58994aa8445f9d02e38ebc9a06d6, State: Running, Role: LEADER
I20260806 16:12:18.650338 21837 consensus_queue.cc:237] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45861 } }
I20260806 16:12:18.651101 21832 sys_catalog.cc:564] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:18.652112 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45861 } } }
I20260806 16:12:18.652297 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:18.652603 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be4f58994aa8445f9d02e38ebc9a06d6. Latest consensus state: current_term: 1 leader_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4f58994aa8445f9d02e38ebc9a06d6" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45861 } } }
I20260806 16:12:18.652706 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:18.652865 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:18.657243 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:18.664130 21848 catalog_manager.cc:1347] Generated new cluster ID: 98e1cc7b1ce941948b2424fb066f4666
I20260806 16:12:18.664211 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:18.672387 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:18.666932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.18.90.62:45861)
0806 16:12:18.667062 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:18.667067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:18.667499 (+ 432us) server_negotiation.cc:407] Connection header received
0806 16:12:18.667715 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:18.667719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:18.667775 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:18.667874 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:18.668526 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:18.669058 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:18.669972 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:18.670237 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:18.670354 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:18.671316 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:18.671320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:18.671323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:18.671815 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:18.671847 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:18.671865 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:18.671975 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:18.672103 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 16:12:18.672170 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:12:18.682308 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:18.683334 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:18.691857 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be4f58994aa8445f9d02e38ebc9a06d6: Generated new TSK 0
I20260806 16:12:18.692729 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:12:18.704797 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21760
2026-08-06T16:12:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1449 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:18 dist-test-slave-0b58 krb5kdc[21866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:18 dist-test-slave-0b58 krb5kdc[21866](info): set up 2 sockets
Aug 06 16:12:18 dist-test-slave-0b58 krb5kdc[21866](info): commencing operation
krb5kdc: starting...
W20260806 16:12:19.765185 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:19 dist-test-slave-0b58 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:19.893491 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36445
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:36445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:20.013423 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:20.013775 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:20.013866 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:20.018344 21880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:20.018460 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:20.018500 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:20.018539 21880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:20.018575 21880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:20.024181 21880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:36445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:20.025409 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:20.026506 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:20.035481 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:20.035471 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.035528 21880 server_base.cc:1034] running on GCE node
W20260806 16:12:20.035471 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.036453 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:20.037601 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:20.038779 21880 hybrid_clock.cc:648] HybridClock initialized: now 1786032740038751 us; error 41 us; skew 500 ppm
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:20.041761 21880 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:20.042759 21880 webserver.cc:460] Webserver started at http://127.18.90.62:33711/ using document root <none> and password file <none>
I20260806 16:12:20.043373 21880 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:20.043470 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:20.043712 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:20.045954 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "5f250cd1b57b488aad7789dc1e1aca32"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "d3f359f395f9ebd9c8d973f87aff4dee"
server_key_iv: "6344b2084dc9111a4dc752fcbf3f9fc4"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.046588 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "5f250cd1b57b488aad7789dc1e1aca32"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "d3f359f395f9ebd9c8d973f87aff4dee"
server_key_iv: "6344b2084dc9111a4dc752fcbf3f9fc4"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.050585 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:20.053313 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.054598 21880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:20.054759 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "5f250cd1b57b488aad7789dc1e1aca32"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "d3f359f395f9ebd9c8d973f87aff4dee"
server_key_iv: "6344b2084dc9111a4dc752fcbf3f9fc4"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.054889 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:20.092171 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:20.096212 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:20.096474 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:20.104039 21880 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:36445
I20260806 16:12:20.104369 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:36445 every 8 connection(s)
I20260806 16:12:20.105207 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:20.108135 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:20.111299 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21880
I20260806 16:12:20.111430 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:20.111739 18792 external_mini_cluster.cc:1428] Setting key f9d973d9bfd3c1f3e2f359d250d567c4
I20260806 16:12:20.114987 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32: Bootstrap starting.
I20260806 16:12:20.117847 21951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:20.118744 21951 log.cc:824] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:20.121353 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32: No bootstrap required, opened a new log
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:20.126003 21951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f250cd1b57b488aad7789dc1e1aca32" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36445 } }
I20260806 16:12:20.126310 21951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:20.126395 21951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f250cd1b57b488aad7789dc1e1aca32, State: Initialized, Role: FOLLOWER
I20260806 16:12:20.126852 21951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f250cd1b57b488aad7789dc1e1aca32" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36445 } }
I20260806 16:12:20.127003 21951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:20.127094 21951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:20.127229 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.113328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.18.90.62:36445)
0806 16:12:20.113787 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:20.113804 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:20.113843 (+ 39us) server_negotiation.cc:407] Connection header received
0806 16:12:20.114488 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:20.114506 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:20.114869 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:20.115232 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:20.116119 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.117211 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.118433 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.118962 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.119083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.122137 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:20.122142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:20.122157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:20.122186 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:20.123954 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.124499 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.124504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.124508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.124625 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.124963 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.124967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.124970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.125387 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:20.125545 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:20.125840 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 16:12:20.126188 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":169,"threads_started":1}
I20260806 16:12:20.127230 21951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:20.128348 21951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f250cd1b57b488aad7789dc1e1aca32" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36445 } }
I20260806 16:12:20.128713 21951 leader_election.cc:302] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f250cd1b57b488aad7789dc1e1aca32; no voters:
I20260806 16:12:20.129091 21951 leader_election.cc:288] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:20.129186 21956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:20.129360 21956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [term 1 LEADER]: Becoming Leader. State: Replica: 5f250cd1b57b488aad7789dc1e1aca32, State: Running, Role: LEADER
I20260806 16:12:20.129702 21956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f250cd1b57b488aad7789dc1e1aca32" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36445 } }
I20260806 16:12:20.130420 21951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:20.131443 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f250cd1b57b488aad7789dc1e1aca32. Latest consensus state: current_term: 1 leader_uuid: "5f250cd1b57b488aad7789dc1e1aca32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f250cd1b57b488aad7789dc1e1aca32" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36445 } } }
I20260806 16:12:20.131708 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f250cd1b57b488aad7789dc1e1aca32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f250cd1b57b488aad7789dc1e1aca32" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36445 } } }
I20260806 16:12:20.131829 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:20.131908 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32 [sys.catalog]: This master's current role is: LEADER
W20260806 16:12:20.135428 21971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:12:20.135861 21971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:12:20.135957 21966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:20.136708 21966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:20.142490 21966 catalog_manager.cc:1347] Generated new cluster ID: 0940e659a43843f29fe90e3639ac99ae
I20260806 16:12:20.142570 21966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:20.162609 21966 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:20.164011 21966 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:20.169554 21966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f250cd1b57b488aad7789dc1e1aca32: Generated new TSK 0
I20260806 16:12:20.170392 21966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:20.212402 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36445
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:20.335505 21977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:20.335816 21977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:20.335861 21977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:20.340046 21977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:20.340121 21977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:20.340183 21977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:20.345780 21977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:20.346988 21977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:20.348102 21977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:20.358692 21985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.358937 21977 server_base.cc:1034] running on GCE node
W20260806 16:12:20.358685 21982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:20.358685 21983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.359591 21977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:20.360337 21977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:20.361541 21977 hybrid_clock.cc:648] HybridClock initialized: now 1786032740361520 us; error 32 us; skew 500 ppm
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:20.364578 21977 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:20.365634 21977 webserver.cc:460] Webserver started at http://127.18.90.1:43625/ using document root <none> and password file <none>
I20260806 16:12:20.366245 21977 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:20.366353 21977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:20.366596 21977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:20.368846 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "5f421cc9f0c94e90b39afafe7cfd81ea"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "5c0ecc7759c84e64ba874f8dbfd2e6ac"
server_key_iv: "0f64633dbddf3d70d3c96ff15d2fce7f"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.369455 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "5f421cc9f0c94e90b39afafe7cfd81ea"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "5c0ecc7759c84e64ba874f8dbfd2e6ac"
server_key_iv: "0f64633dbddf3d70d3c96ff15d2fce7f"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.373484 21977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:20.376085 21992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.377429 21977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:20.377592 21977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "5f421cc9f0c94e90b39afafe7cfd81ea"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "5c0ecc7759c84e64ba874f8dbfd2e6ac"
server_key_iv: "0f64633dbddf3d70d3c96ff15d2fce7f"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.377718 21977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:20.392206 21977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:20.395634 21977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:20.395861 21977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:20.396579 21977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:20.397459 21977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:20.397532 21977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.397603 21977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:20.397645 21977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.407148 21977 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:37439
I20260806 16:12:20.407208 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:37439 every 8 connection(s)
I20260806 16:12:20.408181 21977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:20.410490 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 21977
I20260806 16:12:20.410624 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:20.410976 18792 external_mini_cluster.cc:1428] Setting key 7624e65d73e2644e90ad65a795f8cc86
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:20.425266 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.410312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:42607 (local address 127.18.90.62:36445)
0806 16:12:20.410467 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:20.410471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:20.411625 (+ 1154us) server_negotiation.cc:407] Connection header received
0806 16:12:20.413355 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:20.413362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:20.413429 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:20.413532 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:20.414877 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.415780 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.417172 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.417464 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.417564 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.420960 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:20.420967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:20.420974 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:20.421011 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:20.422820 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.423413 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.423419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.423422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.423516 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.423878 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.423884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.423889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.424177 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:20.424333 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:20.424999 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 16:12:20.425116 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:20.426461 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.410803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36445 (local address 127.18.90.1:42607)
0806 16:12:20.411417 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:20.411486 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:20.413127 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:20.413731 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:20.413741 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:20.414155 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:20.414707 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:20.414723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.415937 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.415940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:20.416989 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:20.416999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.417676 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.417683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:20.417951 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.418703 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:20.418736 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:20.420744 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:20.422964 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:20.422969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:20.422982 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:20.423280 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:20.423614 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:20.423616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:20.423617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:20.423762 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:20.424356 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:20.424362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:20.424669 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 16:12:20.424897 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 16:12:20.425450 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:12:20.428085 22106 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36445
I20260806 16:12:20.428403 22106 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:20.429001 22106 heartbeater.cc:511] Master 127.18.90.62:36445 requested a full tablet report, sending...
I20260806 16:12:20.430981 21912 ts_manager.cc:194] Registered new tserver with Master: 5f421cc9f0c94e90b39afafe7cfd81ea (127.18.90.1:37439)
I20260806 16:12:20.432538 21912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:42607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:20.448815 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36445
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:20.567700 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:20.567982 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:20.568064 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:20.572177 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:20.572283 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:20.572346 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:20.577968 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:20.579176 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:20.580315 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:20.590633 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.590785 22111 server_base.cc:1034] running on GCE node
W20260806 16:12:20.590633 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:20.590685 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.591490 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:20.592190 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:20.593420 22111 hybrid_clock.cc:648] HybridClock initialized: now 1786032740593384 us; error 49 us; skew 500 ppm
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:20.596580 22111 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:20.597522 22111 webserver.cc:460] Webserver started at http://127.18.90.2:46257/ using document root <none> and password file <none>
I20260806 16:12:20.598124 22111 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:20.598237 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:20.598489 22111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:20.600857 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "a881de3ec1d845e2838c91a4f266e2a6"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "3ddd4621e1a7ee04d36d28c27fc83861"
server_key_iv: "2d65d97759c4b33531e5efd6d77f665b"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.601549 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "a881de3ec1d845e2838c91a4f266e2a6"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "3ddd4621e1a7ee04d36d28c27fc83861"
server_key_iv: "2d65d97759c4b33531e5efd6d77f665b"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.605541 22111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:20.608139 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.609551 22111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:20.609709 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "a881de3ec1d845e2838c91a4f266e2a6"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "3ddd4621e1a7ee04d36d28c27fc83861"
server_key_iv: "2d65d97759c4b33531e5efd6d77f665b"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.609843 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:20.641431 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:20.645057 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:20.645300 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:20.645987 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:20.646899 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:20.646975 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.647068 22111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:20.647113 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.656759 22111 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:39193
I20260806 16:12:20.656787 22243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:39193 every 8 connection(s)
I20260806 16:12:20.657760 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:20.667214 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22111
I20260806 16:12:20.667323 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:20.667659 18792 external_mini_cluster.cc:1428] Setting key 17f76c0bcb8dc42ef94702e855e2124b
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:20.673447 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.659802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:35729 (local address 127.18.90.62:36445)
0806 16:12:20.659946 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:20.659950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:20.660933 (+ 983us) server_negotiation.cc:407] Connection header received
0806 16:12:20.662406 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:20.662410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:20.662473 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:20.662555 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:20.663814 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.664463 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.665678 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.665931 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.666063 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.668973 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:20.668976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:20.668978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:20.669003 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:20.670709 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.671348 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.671353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.671355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.671446 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.671910 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.671916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.671920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.672241 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:20.672399 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:20.673204 (+ 805us) server_negotiation.cc:299] Negotiation successful
0806 16:12:20.673321 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:12:20.674789 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.660222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36445 (local address 127.18.90.2:35729)
0806 16:12:20.660756 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:20.660805 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:20.662160 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:20.662739 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:20.662746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:20.663125 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:20.663641 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:20.663655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.664602 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.664610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:20.665538 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:20.665548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.666112 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.666116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:20.666315 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.666926 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:20.666943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:20.668773 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:20.670861 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:20.670867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:20.670885 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:20.671206 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:20.671547 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:20.671551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:20.671553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:20.671772 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:20.672420 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:20.672430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:20.672917 (+ 487us) client_negotiation.cc:769] Sending connection context
0806 16:12:20.673143 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 16:12:20.673545 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260806 16:12:20.676165 22244 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36445
I20260806 16:12:20.676457 22244 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:20.676990 22244 heartbeater.cc:511] Master 127.18.90.62:36445 requested a full tablet report, sending...
I20260806 16:12:20.678185 21912 ts_manager.cc:194] Registered new tserver with Master: a881de3ec1d845e2838c91a4f266e2a6 (127.18.90.2:39193)
I20260806 16:12:20.678798 21912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:35729
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:20.697557 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36445
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:20.814913 22249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:20.815326 22249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:20.815402 22249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:20.819519 22249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:20.819587 22249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:20.819648 22249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:20.825184 22249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:20.826328 22249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:20.827361 22249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:20.837684 22255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:20.837700 22254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:20.837708 22257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:20.837779 22249 server_base.cc:1034] running on GCE node
I20260806 16:12:20.838408 22249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:20.839030 22249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:20.840235 22249 hybrid_clock.cc:648] HybridClock initialized: now 1786032740840221 us; error 27 us; skew 500 ppm
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:20.843170 22249 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:20.844138 22249 webserver.cc:460] Webserver started at http://127.18.90.3:39783/ using document root <none> and password file <none>
I20260806 16:12:20.844745 22249 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:20.844815 22249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:20.844985 22249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:20.847165 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "b5277c0ac13c4a90a39adc1a28b4e4ad"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "af6512087fe60c9532290864d889c182"
server_key_iv: "75fb7123d726a88b915b99298119f556"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.847731 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "b5277c0ac13c4a90a39adc1a28b4e4ad"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "af6512087fe60c9532290864d889c182"
server_key_iv: "75fb7123d726a88b915b99298119f556"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.851621 22249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:20.854410 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.855754 22249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:20.855896 22249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "b5277c0ac13c4a90a39adc1a28b4e4ad"
format_stamp: "Formatted at 2026-08-06 16:12:20 on dist-test-slave-0b58"
server_key: "af6512087fe60c9532290864d889c182"
server_key_iv: "75fb7123d726a88b915b99298119f556"
server_key_version: "encryptionkey@0"
I20260806 16:12:20.856030 22249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:20.867811 22249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:20.871371 22249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:20.871595 22249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:20.872231 22249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:20.873214 22249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:20.873293 22249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.873366 22249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:20.873418 22249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:20.882753 22249 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:46215
I20260806 16:12:20.882781 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:46215 every 8 connection(s)
I20260806 16:12:20.883740 22249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:20.884989 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22249
I20260806 16:12:20.885119 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:20.885442 18792 external_mini_cluster.cc:1428] Setting key 854f382255cc26bf1803224ef2a3eba8
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:20.898546 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.885861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:50715 (local address 127.18.90.62:36445)
0806 16:12:20.886020 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:20.886023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:20.887106 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 16:12:20.888593 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:20.888595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:20.888641 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:20.888722 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:20.889924 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.890458 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.891642 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.891899 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.892000 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.894606 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:20.894609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:20.894611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:20.894638 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:20.896179 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.896768 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.896775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.896779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.896887 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.897267 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.897271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.897272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.897475 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:20.897558 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:20.898223 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 16:12:20.898364 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:20.899648 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.886428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36445 (local address 127.18.90.3:50715)
0806 16:12:20.886949 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:20.887004 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:20.888396 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:20.888900 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:20.888906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:20.889282 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:20.889789 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:20.889803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.890580 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.890584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:20.891498 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:20.891508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.892052 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.892055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:20.892206 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.892865 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:20.892882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:20.894406 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:20.896346 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:20.896351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:20.896364 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:20.896637 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:20.897005 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:20.897008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:20.897010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:20.897166 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:20.897575 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:20.897581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:20.897918 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 16:12:20.898123 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 16:12:20.898496 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260806 16:12:20.900878 22379 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36445
I20260806 16:12:20.901141 22379 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:20.901706 22379 heartbeater.cc:511] Master 127.18.90.62:36445 requested a full tablet report, sending...
I20260806 16:12:20.902824 21912 ts_manager.cc:194] Registered new tserver with Master: b5277c0ac13c4a90a39adc1a28b4e4ad (127.18.90.3:46215)
I20260806 16:12:20.903385 21912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:50715
I20260806 16:12:20.912127 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 16:12:20 dist-test-slave-0b58 krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:20.939441 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:20.930930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.18.90.62:36445)
0806 16:12:20.931094 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:20.931100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:20.931294 (+ 194us) server_negotiation.cc:407] Connection header received
0806 16:12:20.931455 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:20.931458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:20.931507 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:20.931581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:20.932372 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.932912 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.933762 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:20.933979 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:20.934064 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:20.936526 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:20.936529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:20.936531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:20.936558 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:20.937872 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.938261 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.938263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.938265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.938336 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:20.938605 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:20.938608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:20.938609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:20.938864 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:20.938983 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:20.939184 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 16:12:20.939300 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 16:12:20.940006 21912 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52108
I20260806 16:12:20.941203 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21977
I20260806 16:12:20.948849 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22111
I20260806 16:12:20.955556 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22249
I20260806 16:12:20.962052 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 21880
2026-08-06T16:12:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2260 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:21 dist-test-slave-0b58 krb5kdc[22391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:21 dist-test-slave-0b58 krb5kdc[22391](info): set up 2 sockets
Aug 06 16:12:21 dist-test-slave-0b58 krb5kdc[22391](info): commencing operation
krb5kdc: starting...
W20260806 16:12:22.013787 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:22.141368 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42163
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:22.258970 22404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:22.259248 22404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:22.259291 22404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:22.263432 22404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:22.263500 22404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:22.263521 22404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:22.263538 22404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:22.263594 22404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:22.269095 22404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:22.270247 22404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:22.271349 22404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:12:22.280247 22404 server_base.cc:1034] running on GCE node
W20260806 16:12:22.280292 22412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:22.280298 22409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:22.280445 22410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:22.281129 22404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:22.282282 22404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:22.283480 22404 hybrid_clock.cc:648] HybridClock initialized: now 1786032742283451 us; error 42 us; skew 500 ppm
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:22.286413 22404 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:22.287350 22404 webserver.cc:460] Webserver started at http://127.18.90.62:35425/ using document root <none> and password file <none>
I20260806 16:12:22.287933 22404 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:22.288028 22404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:22.288306 22404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:22.290403 22404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "0ca2c335480d00016c1652ed7a8d6ee9"
server_key_iv: "4223a10730e2c31728b31a3424ec81b5"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.290984 22404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "0ca2c335480d00016c1652ed7a8d6ee9"
server_key_iv: "4223a10730e2c31728b31a3424ec81b5"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.294807 22404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:22.297192 22420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.298520 22404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:22.298666 22404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "0ca2c335480d00016c1652ed7a8d6ee9"
server_key_iv: "4223a10730e2c31728b31a3424ec81b5"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.298785 22404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:22.324452 22404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:22.327692 22404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:22.327934 22404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:22.335502 22404 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:42163
I20260806 16:12:22.335530 22474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:42163 every 8 connection(s)
I20260806 16:12:22.336615 22404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:22.338749 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22404
I20260806 16:12:22.338900 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:22.339206 18792 external_mini_cluster.cc:1428] Setting key 2688e91f62272a2b463c78c750a744c3
I20260806 16:12:22.339826 22475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:22.346035 22475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da: Bootstrap starting.
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:22.348421 22475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:22.349198 22475 log.cc:824] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:22.351181 22475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da: No bootstrap required, opened a new log
I20260806 16:12:22.353574 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:22.340744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.18.90.62:42163)
0806 16:12:22.341226 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:22.341243 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:22.341298 (+ 55us) server_negotiation.cc:407] Connection header received
0806 16:12:22.341979 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:22.341996 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:22.342348 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:22.342694 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:22.343548 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.344639 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.345530 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.346044 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.346230 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:22.348801 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:22.348806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:22.348820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:22.348848 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:22.350639 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:22.351113 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:22.351118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:22.351123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:22.351280 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:22.351589 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:22.351592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:22.351594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:22.351999 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:22.352215 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:22.352511 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 16:12:22.352817 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
I20260806 16:12:22.355039 22475 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42163 } }
I20260806 16:12:22.355331 22475 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:22.355400 22475 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b9874e6f124ad8bc8b6f7fe0b9f8da, State: Initialized, Role: FOLLOWER
I20260806 16:12:22.355885 22475 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42163 } }
I20260806 16:12:22.356047 22475 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:22.356119 22475 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:22.356195 22475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:22.357168 22475 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42163 } }
I20260806 16:12:22.357482 22475 leader_election.cc:302] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b9874e6f124ad8bc8b6f7fe0b9f8da; no voters:
I20260806 16:12:22.357743 22475 leader_election.cc:288] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:22.357887 22480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:22.358155 22480 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [term 1 LEADER]: Becoming Leader. State: Replica: f2b9874e6f124ad8bc8b6f7fe0b9f8da, State: Running, Role: LEADER
I20260806 16:12:22.358538 22480 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42163 } }
I20260806 16:12:22.359156 22475 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:22.360389 22482 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b9874e6f124ad8bc8b6f7fe0b9f8da. Latest consensus state: current_term: 1 leader_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42163 } } }
I20260806 16:12:22.360529 22482 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:22.360816 22481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9874e6f124ad8bc8b6f7fe0b9f8da" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42163 } } }
I20260806 16:12:22.360924 22481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:22.360926 22489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:22.364225 22489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:22.370664 22489 catalog_manager.cc:1347] Generated new cluster ID: 054f8268207844e283207ec5ef642263
I20260806 16:12:22.370766 22489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:22.389652 22489 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:22.390703 22489 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:22.397898 22489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b9874e6f124ad8bc8b6f7fe0b9f8da: Generated new TSK 0
I20260806 16:12:22.398737 22489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:22.435124 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42163
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:22.579978 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:22.580354 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:22.580473 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:22.585050 22501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:22.585162 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:22.585283 22501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:22.591038 22501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:22.592234 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:22.593415 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:22.603199 22507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:22.603212 22506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:22.603377 22501 server_base.cc:1034] running on GCE node
W20260806 16:12:22.603216 22509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:22.603966 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:22.604729 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:22.605924 22501 hybrid_clock.cc:648] HybridClock initialized: now 1786032742605906 us; error 28 us; skew 500 ppm
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:22.608832 22501 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:22.609750 22501 webserver.cc:460] Webserver started at http://127.18.90.1:41647/ using document root <none> and password file <none>
I20260806 16:12:22.610360 22501 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:22.610473 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:22.610767 22501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:22.613164 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "490ca952249f45c4bf96afb613a265a7"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "b2732c5514514991158fb17b45daab21"
server_key_iv: "ee6aea299fb0c702439a787e0310a162"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.613765 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "490ca952249f45c4bf96afb613a265a7"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "b2732c5514514991158fb17b45daab21"
server_key_iv: "ee6aea299fb0c702439a787e0310a162"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.617660 22501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:22.620119 22516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.621456 22501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:22.621598 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "490ca952249f45c4bf96afb613a265a7"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "b2732c5514514991158fb17b45daab21"
server_key_iv: "ee6aea299fb0c702439a787e0310a162"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.621721 22501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:22.640071 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:22.643381 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:22.643626 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:22.644309 22501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:22.645205 22501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:22.645283 22501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.645359 22501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:22.645402 22501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.654651 22501 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:45983
I20260806 16:12:22.654716 22631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:45983 every 8 connection(s)
I20260806 16:12:22.655630 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:22.664165 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22501
I20260806 16:12:22.664345 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:22.664721 18792 external_mini_cluster.cc:1428] Setting key 9859067f3e7b63bb3fa59b516ff0810b
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:22.670857 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:22.657620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:48371 (local address 127.18.90.62:42163)
0806 16:12:22.657844 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:22.657848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:22.658666 (+ 818us) server_negotiation.cc:407] Connection header received
0806 16:12:22.660237 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:22.660241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:22.660335 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:22.660421 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:22.661729 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.662264 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.663441 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.663666 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.663757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:22.666576 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:22.666579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:22.666582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:22.666614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:22.668429 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:22.669148 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:22.669152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:22.669153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:22.669235 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:22.669645 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:22.669647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:22.669648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:22.669841 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:22.669919 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:22.670604 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 16:12:22.670719 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 16:12:22.672295 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:22.658020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42163 (local address 127.18.90.1:48371)
0806 16:12:22.658498 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:22.658547 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:22.660001 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:22.660604 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:22.660611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:22.660985 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:22.661582 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:22.661596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.662400 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.662404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:22.663324 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:22.663334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.663807 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.663810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:22.664056 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:22.664750 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:22.664774 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:22.666397 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:22.668574 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:22.668583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:22.668604 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:22.669029 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:22.669331 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:22.669336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:22.669340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:22.669547 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:22.669952 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:22.669963 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:22.670337 (+ 374us) client_negotiation.cc:769] Sending connection context
0806 16:12:22.670571 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 16:12:22.670987 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260806 16:12:22.673717 22632 heartbeater.cc:347] Connected to a master server at 127.18.90.62:42163
I20260806 16:12:22.673990 22632 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:22.674526 22632 heartbeater.cc:511] Master 127.18.90.62:42163 requested a full tablet report, sending...
I20260806 16:12:22.676167 22437 ts_manager.cc:194] Registered new tserver with Master: 490ca952249f45c4bf96afb613a265a7 (127.18.90.1:45983)
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:22.677589 22437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:48371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:22.696017 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42163
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:22.812160 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:22.812538 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:22.812587 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:22.816680 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:22.816744 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:22.816807 22637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:22.822126 22637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:22.823285 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:22.824357 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:22.834396 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:22.834508 22637 server_base.cc:1034] running on GCE node
W20260806 16:12:22.834404 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:22.834396 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:22.835175 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:22.835839 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:22.837028 22637 hybrid_clock.cc:648] HybridClock initialized: now 1786032742837012 us; error 27 us; skew 500 ppm
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:22.839922 22637 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:22.840898 22637 webserver.cc:460] Webserver started at http://127.18.90.2:38565/ using document root <none> and password file <none>
I20260806 16:12:22.841449 22637 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:22.841512 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:22.841681 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:22.843758 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "cc60549d32cc4190808c25a52dcc8a33"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "cd667792ceb40513455458f0a8a305f5"
server_key_iv: "2462be90afad35eec75f8487d3998777"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.844364 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "cc60549d32cc4190808c25a52dcc8a33"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "cd667792ceb40513455458f0a8a305f5"
server_key_iv: "2462be90afad35eec75f8487d3998777"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.848134 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:12:22.850857 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.852139 22637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:22.852308 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "cc60549d32cc4190808c25a52dcc8a33"
format_stamp: "Formatted at 2026-08-06 16:12:22 on dist-test-slave-0b58"
server_key: "cd667792ceb40513455458f0a8a305f5"
server_key_iv: "2462be90afad35eec75f8487d3998777"
server_key_version: "encryptionkey@0"
I20260806 16:12:22.852422 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:22.863411 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:22.866844 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:22.867061 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:22.867689 22637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:22.868602 22637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:22.868654 22637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.868722 22637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:22.868760 22637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:22.878300 22637 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:36519
I20260806 16:12:22.878327 22766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:36519 every 8 connection(s)
I20260806 16:12:22.879223 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:22.883613 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22637
I20260806 16:12:22.883754 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:22.884065 18792 external_mini_cluster.cc:1428] Setting key e74c5db8e49e2f396f7e72da82892fdf
Aug 06 16:12:22 dist-test-slave-0b58 krb5kdc[22391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:22.894416 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:22.881276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:38289 (local address 127.18.90.62:42163)
0806 16:12:22.881441 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:22.881446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:22.882344 (+ 898us) server_negotiation.cc:407] Connection header received
0806 16:12:22.883740 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:22.883743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:22.883803 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:22.883892 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:22.885351 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.885885 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.887098 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.887375 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.887492 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:22.890203 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:22.890209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:22.890216 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:22.890261 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:22.892035 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:22.892620 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:22.892623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:22.892625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:22.892726 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:22.893075 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:22.893079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:22.893082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:22.893373 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:22.893488 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:22.894016 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 16:12:22.894160 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:12:22.895151 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:22.881706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42163 (local address 127.18.90.2:38289)
0806 16:12:22.882168 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:22.882214 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:22.883518 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:22.884090 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:22.884097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:22.884679 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:22.885198 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:22.885212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.886023 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.886027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:22.886959 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:22.886969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:22.887658 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:22.887661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:22.887813 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:22.888486 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:22.888503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:22.889954 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:22.892164 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:22.892168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:22.892183 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:22.892510 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:22.892822 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:22.892824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:22.892825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:22.892967 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:22.893481 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:22.893486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:22.893733 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 16:12:22.893895 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 16:12:22.894168 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260806 16:12:22.896487 22767 heartbeater.cc:347] Connected to a master server at 127.18.90.62:42163
I20260806 16:12:22.896776 22767 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:22.897383 22767 heartbeater.cc:511] Master 127.18.90.62:42163 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:12:22.898695 22437 ts_manager.cc:194] Registered new tserver with Master: cc60549d32cc4190808c25a52dcc8a33 (127.18.90.2:36519)
I20260806 16:12:22.899597 22437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:38289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:22.914649 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42163
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:23.031258 22772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:23.031519 22772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:23.031607 22772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:23.035830 22772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:23.035907 22772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:23.036003 22772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:23.041666 22772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:23.042855 22772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:23.043954 22772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:23.054070 22777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:23.054108 22778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:23.054085 22772 server_base.cc:1034] running on GCE node
W20260806 16:12:23.054100 22780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:23.054808 22772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:23.055488 22772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:23.056764 22772 hybrid_clock.cc:648] HybridClock initialized: now 1786032743056746 us; error 28 us; skew 500 ppm
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:23.059612 22772 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:23.060575 22772 webserver.cc:460] Webserver started at http://127.18.90.3:39225/ using document root <none> and password file <none>
I20260806 16:12:23.061252 22772 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:23.061318 22772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:23.061529 22772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:23.063652 22772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "e6b8eb20a73a46039f7364b53adf099f"
format_stamp: "Formatted at 2026-08-06 16:12:23 on dist-test-slave-0b58"
server_key: "9fe3428d444508b3ab87b133fc089490"
server_key_iv: "13f667ec4b9d9baa39e03e67d99e1824"
server_key_version: "encryptionkey@0"
I20260806 16:12:23.064213 22772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "e6b8eb20a73a46039f7364b53adf099f"
format_stamp: "Formatted at 2026-08-06 16:12:23 on dist-test-slave-0b58"
server_key: "9fe3428d444508b3ab87b133fc089490"
server_key_iv: "13f667ec4b9d9baa39e03e67d99e1824"
server_key_version: "encryptionkey@0"
I20260806 16:12:23.067988 22772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:23.070540 22787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:23.071873 22772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:12:23.072012 22772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "e6b8eb20a73a46039f7364b53adf099f"
format_stamp: "Formatted at 2026-08-06 16:12:23 on dist-test-slave-0b58"
server_key: "9fe3428d444508b3ab87b133fc089490"
server_key_iv: "13f667ec4b9d9baa39e03e67d99e1824"
server_key_version: "encryptionkey@0"
I20260806 16:12:23.072125 22772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:23.083765 22772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:23.086910 22772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:23.087119 22772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:23.087705 22772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:23.088569 22772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:23.088617 22772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:23.088658 22772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:23.088675 22772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:23.097564 22772 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:41481
I20260806 16:12:23.097585 22901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:41481 every 8 connection(s)
I20260806 16:12:23.098508 22772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:23.102020 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22772
I20260806 16:12:23.102173 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:23.102536 18792 external_mini_cluster.cc:1428] Setting key b5c968a76e6f229981ad9b19d622beba
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:23.113423 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:23.100587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:59289 (local address 127.18.90.62:42163)
0806 16:12:23.100758 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:23.100763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:23.101625 (+ 862us) server_negotiation.cc:407] Connection header received
0806 16:12:23.103399 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:23.103402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:23.103493 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:23.103600 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:23.105065 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.105587 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.106794 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.107030 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.107138 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:23.109695 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:23.109698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:23.109700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:23.109726 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:23.111282 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:23.111773 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:23.111776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:23.111777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:23.111856 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:23.112184 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:23.112209 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:23.112210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:23.112551 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:23.112663 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:23.113183 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 16:12:23.113296 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:12:23.114270 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:23.101009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42163 (local address 127.18.90.3:59289)
0806 16:12:23.101467 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:23.101512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:23.103160 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:23.103789 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:23.103797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:23.104317 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:23.104914 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:23.104929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.105717 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.105722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:23.106657 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:23.106667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.107202 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.107207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:23.107374 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:23.107957 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:23.107974 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:23.109528 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:23.111391 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:23.111395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:23.111407 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:23.111670 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:23.111940 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:23.111942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:23.111943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:23.112075 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:23.112658 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:23.112664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:23.112911 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 16:12:23.113075 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 16:12:23.113343 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260806 16:12:23.115500 22902 heartbeater.cc:347] Connected to a master server at 127.18.90.62:42163
I20260806 16:12:23.115763 22902 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:23.116334 22902 heartbeater.cc:511] Master 127.18.90.62:42163 requested a full tablet report, sending...
I20260806 16:12:23.117472 22437 ts_manager.cc:194] Registered new tserver with Master: e6b8eb20a73a46039f7364b53adf099f (127.18.90.3:41481)
I20260806 16:12:23.118007 22437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:59289
I20260806 16:12:23.128679 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:23.156816 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:23.146434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.18.90.1:45983)
0806 16:12:23.146769 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:23.146785 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:23.146809 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:12:23.146865 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:23.146870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:23.147059 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:23.147184 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:23.147962 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.148876 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.149762 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.150154 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.150252 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:23.152866 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:23.152878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:23.152893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:23.152924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:23.154709 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:23.155164 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:23.155171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:23.155172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:23.155275 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:23.155584 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:23.155586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:23.155587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:23.155919 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:23.156112 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:23.156391 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 16:12:23.156570 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:23.168864 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:23.160077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.18.90.62:42163)
0806 16:12:23.160253 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:23.160258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:23.160468 (+ 210us) server_negotiation.cc:407] Connection header received
0806 16:12:23.160619 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:23.160622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:23.160684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:23.160773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:23.161450 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.161991 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.162797 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:23.163026 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:23.163137 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:23.165747 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:23.165750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:23.165752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:23.165780 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:23.167083 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:23.167516 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:23.167520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:23.167522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:23.167609 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:23.167885 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:23.167889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:23.167891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:23.168131 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:23.168327 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:23.168554 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 16:12:23.168685 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 16:12:23.169366 22437 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33048
I20260806 16:12:23.170044 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22501
I20260806 16:12:23.177098 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22637
I20260806 16:12:23.183059 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22772
I20260806 16:12:23.189260 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22404
2026-08-06T16:12:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2225 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22914](info): set up 2 sockets
Aug 06 16:12:23 dist-test-slave-0b58 krb5kdc[22914](info): commencing operation
krb5kdc: starting...
W20260806 16:12:24.231501 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:24 dist-test-slave-0b58 krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:24.359922 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42129
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:24.473673 22927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:24.473969 22927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:24.474013 22927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:24.478052 22927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:24.478114 22927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:24.478134 22927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:24.478152 22927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:24.478166 22927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:24.483417 22927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:24.484539 22927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:24.485559 22927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:24.496055 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:24.496048 22932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:24.496047 22933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:24.497231 22927 server_base.cc:1034] running on GCE node
I20260806 16:12:24.498027 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:24.499317 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:24.500489 22927 hybrid_clock.cc:648] HybridClock initialized: now 1786032744500453 us; error 50 us; skew 500 ppm
Aug 06 16:12:24 dist-test-slave-0b58 krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:24.510007 22927 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:24.510911 22927 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:24.511014 22927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:24.511270 22927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:24.513553 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "8fad60320bf04854a4920c1f379f23cd"
format_stamp: "Formatted at 2026-08-06 16:12:24 on dist-test-slave-0b58"
server_key: "3013e118736cca092579ce0d80357a45"
server_key_iv: "4125717a7916f52d275b3113a7672166"
server_key_version: "encryptionkey@0"
I20260806 16:12:24.514174 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "8fad60320bf04854a4920c1f379f23cd"
format_stamp: "Formatted at 2026-08-06 16:12:24 on dist-test-slave-0b58"
server_key: "3013e118736cca092579ce0d80357a45"
server_key_iv: "4125717a7916f52d275b3113a7672166"
server_key_version: "encryptionkey@0"
I20260806 16:12:24.518210 22927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:24.520752 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:24.522050 22927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:24.522200 22927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "8fad60320bf04854a4920c1f379f23cd"
format_stamp: "Formatted at 2026-08-06 16:12:24 on dist-test-slave-0b58"
server_key: "3013e118736cca092579ce0d80357a45"
server_key_iv: "4125717a7916f52d275b3113a7672166"
server_key_version: "encryptionkey@0"
I20260806 16:12:24.522324 22927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:24.541639 22927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:24.544808 22927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:24.545024 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:24.552465 22927 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:42129
I20260806 16:12:24.552724 22994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:42129 every 8 connection(s)
I20260806 16:12:24.553184 22927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:24.556133 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:24.557191 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 22927
I20260806 16:12:24.557304 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:24.557615 18792 external_mini_cluster.cc:1428] Setting key 1a39cb325946e0230f53e427aa1f506f
I20260806 16:12:24.563436 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd: Bootstrap starting.
I20260806 16:12:24.566159 22995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:24.566931 22995 log.cc:824] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:12:24 dist-test-slave-0b58 krb5kdc[22914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:24.569393 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd: No bootstrap required, opened a new log
I20260806 16:12:24.573117 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:24.559119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.18.90.62:42129)
0806 16:12:24.559583 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:24.559603 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:24.559661 (+ 58us) server_negotiation.cc:407] Connection header received
0806 16:12:24.560519 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:24.560544 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:24.560984 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:24.561359 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:24.562074 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:24.563578 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:24.564540 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:24.565248 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:24.565411 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:24.568107 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:24.568113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:24.568131 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:24.568160 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:24.570125 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:24.570564 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:24.570570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:24.570576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:24.570713 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:24.570971 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:24.570977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:24.570979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:24.571501 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:24.571726 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:24.571945 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 16:12:24.572359 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":107,"threads_started":1}
I20260806 16:12:24.573783 22995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad60320bf04854a4920c1f379f23cd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42129 } }
I20260806 16:12:24.574039 22995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:24.574116 22995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fad60320bf04854a4920c1f379f23cd, State: Initialized, Role: FOLLOWER
I20260806 16:12:24.574558 22995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad60320bf04854a4920c1f379f23cd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42129 } }
I20260806 16:12:24.574679 22995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:24.574764 22995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:24.574894 22995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:24.575843 22995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad60320bf04854a4920c1f379f23cd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42129 } }
I20260806 16:12:24.576192 22995 leader_election.cc:302] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fad60320bf04854a4920c1f379f23cd; no voters:
I20260806 16:12:24.576586 22995 leader_election.cc:288] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:24.576713 23000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:24.576951 23000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [term 1 LEADER]: Becoming Leader. State: Replica: 8fad60320bf04854a4920c1f379f23cd, State: Running, Role: LEADER
I20260806 16:12:24.577250 23000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad60320bf04854a4920c1f379f23cd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42129 } }
I20260806 16:12:24.577857 22995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:24.579061 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fad60320bf04854a4920c1f379f23cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad60320bf04854a4920c1f379f23cd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42129 } } }
I20260806 16:12:24.579210 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:24.579527 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fad60320bf04854a4920c1f379f23cd. Latest consensus state: current_term: 1 leader_uuid: "8fad60320bf04854a4920c1f379f23cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad60320bf04854a4920c1f379f23cd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42129 } } }
I20260806 16:12:24.579636 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:24.579715 23009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:24.582754 23009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:24.588410 23009 catalog_manager.cc:1347] Generated new cluster ID: 4a8d64c3ec8e43a68a1449d4e27c27f4
I20260806 16:12:24.588487 23009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:24.606199 23009 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:24.607575 23009 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:24.615448 23009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd: Generated new TSK 0
I20260806 16:12:24.616353 23009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:24.651057 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42129
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:24.768113 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:24.768462 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:24.768515 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:24.772558 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:24.772622 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:24.772683 23021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:24.778148 23021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:24.779362 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:24.780490 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:24.790251 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:24.790282 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:24.790268 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:24.790518 23021 server_base.cc:1034] running on GCE node
I20260806 16:12:24.791141 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:24.791870 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:24.793071 23021 hybrid_clock.cc:648] HybridClock initialized: now 1786032744793051 us; error 54 us; skew 500 ppm
Aug 06 16:12:24 dist-test-slave-0b58 krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:24.795961 23021 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:24.796936 23021 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:24.797031 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:24.797259 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:24.799394 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "cedbc1179c54469c927ad969e42d6906"
format_stamp: "Formatted at 2026-08-06 16:12:24 on dist-test-slave-0b58"
server_key: "46ca13f68eb21b03dc05fd4e6e26ed74"
server_key_iv: "62cb8e26669944fdc49f2cf71037c156"
server_key_version: "encryptionkey@0"
I20260806 16:12:24.799918 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "cedbc1179c54469c927ad969e42d6906"
format_stamp: "Formatted at 2026-08-06 16:12:24 on dist-test-slave-0b58"
server_key: "46ca13f68eb21b03dc05fd4e6e26ed74"
server_key_iv: "62cb8e26669944fdc49f2cf71037c156"
server_key_version: "encryptionkey@0"
I20260806 16:12:24.803987 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:24.806573 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:24.807916 23021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:24.808055 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "cedbc1179c54469c927ad969e42d6906"
format_stamp: "Formatted at 2026-08-06 16:12:24 on dist-test-slave-0b58"
server_key: "46ca13f68eb21b03dc05fd4e6e26ed74"
server_key_iv: "62cb8e26669944fdc49f2cf71037c156"
server_key_version: "encryptionkey@0"
I20260806 16:12:24.808182 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:24.826792 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:24.830277 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:24.830492 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:24.831131 23021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:24.832016 23021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:24.832063 23021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:24.832136 23021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:24.832176 23021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:24.841421 23021 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:35757
I20260806 16:12:24.841439 23150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:35757 every 8 connection(s)
I20260806 16:12:24.842074 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:24.848702 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23021
I20260806 16:12:24.848811 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:24.849087 18792 external_mini_cluster.cc:1428] Setting key 6ce039dca4983129f62fd764440cc75e
Aug 06 16:12:24 dist-test-slave-0b58 krb5kdc[22914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:24.857651 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:24.843923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:45963 (local address 127.18.90.62:42129)
0806 16:12:24.844074 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:24.844078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:24.844965 (+ 887us) server_negotiation.cc:407] Connection header received
0806 16:12:24.846433 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:24.846437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:24.846502 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:24.846613 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:24.847847 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:24.848483 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:24.849940 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:24.850203 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:24.850299 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:24.853252 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:24.853256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:24.853258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:24.853285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:24.855145 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:24.855700 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:24.855707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:24.855712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:24.855820 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:24.856202 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:24.856208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:24.856211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:24.856530 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:24.856647 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:24.857228 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 16:12:24.857393 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:12:24.858433 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:24.844316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42129 (local address 127.18.90.1:45963)
0806 16:12:24.844791 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:24.844838 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:24.846201 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:24.846804 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:24.846812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:24.847180 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:24.847690 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:24.847704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:24.848615 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:24.848625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:24.849782 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:24.849793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:24.850391 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:24.850394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:24.850619 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:24.851356 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:24.851372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:24.853054 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:24.855299 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:24.855304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:24.855318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:24.855586 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:24.855942 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:24.855945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:24.855949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:24.856100 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:24.856660 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:24.856666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:24.856939 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 16:12:24.857121 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 16:12:24.857436 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260806 16:12:24.859838 23151 heartbeater.cc:347] Connected to a master server at 127.18.90.62:42129
I20260806 16:12:24.860124 23151 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:24.860813 23151 heartbeater.cc:511] Master 127.18.90.62:42129 requested a full tablet report, sending...
I20260806 16:12:24.862775 22959 ts_manager.cc:194] Registered new tserver with Master: cedbc1179c54469c927ad969e42d6906 (127.18.90.1:35757)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:12:24.864372 22959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:45963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:24.879771 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42129
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:25.001621 23156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:25.001905 23156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:25.002000 23156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:25.006300 23156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:25.006369 23156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:25.006485 23156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:25.012149 23156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:25.013406 23156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:25.014485 23156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:25.024173 23162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:25.024168 23161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:25.024384 23156 server_base.cc:1034] running on GCE node
W20260806 16:12:25.024533 23164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:25.024967 23156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:25.025735 23156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:25.026919 23156 hybrid_clock.cc:648] HybridClock initialized: now 1786032745026903 us; error 28 us; skew 500 ppm
Aug 06 16:12:25 dist-test-slave-0b58 krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:25.029750 23156 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:25.030514 23156 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:25.030614 23156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:25.030827 23156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:25.033020 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "f3c36a637d24486898e2acff23ad261f"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "eb415f8bc6092ea8dec573c9b4551455"
server_key_iv: "b3bf94de7d5d2f7e6b3a42ebb68b2fd0"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.033614 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c36a637d24486898e2acff23ad261f"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "eb415f8bc6092ea8dec573c9b4551455"
server_key_iv: "b3bf94de7d5d2f7e6b3a42ebb68b2fd0"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.037441 23156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:25.039845 23170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.041162 23156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:12:25.041306 23156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "f3c36a637d24486898e2acff23ad261f"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "eb415f8bc6092ea8dec573c9b4551455"
server_key_iv: "b3bf94de7d5d2f7e6b3a42ebb68b2fd0"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.041440 23156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:25.055521 23156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:25.058609 23156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:25.058813 23156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:25.059417 23156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:25.060250 23156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:25.060376 23156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.060468 23156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:25.060520 23156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.069522 23156 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38169
I20260806 16:12:25.069607 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38169 every 8 connection(s)
I20260806 16:12:25.070165 23156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:25.076732 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23156
I20260806 16:12:25.076831 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:25.077121 18792 external_mini_cluster.cc:1428] Setting key c16b75a1ec230482f4ef59e39e7f3e7f
Aug 06 16:12:25 dist-test-slave-0b58 krb5kdc[22914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:25.084875 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.072000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:37229 (local address 127.18.90.62:42129)
0806 16:12:25.072148 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.072151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.073558 (+ 1407us) server_negotiation.cc:407] Connection header received
0806 16:12:25.074489 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.074492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.074554 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.074658 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:25.075851 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.076460 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.077953 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.078261 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.078386 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.081171 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:25.081174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:25.081176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:25.081201 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:25.082710 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:25.083245 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:25.083249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:25.083251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:25.083338 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:25.083698 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:25.083702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:25.083704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:25.083898 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:25.084012 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.084570 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.084708 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:12:25.085969 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.072606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42129 (local address 127.18.90.2:37229)
0806 16:12:25.073403 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.073446 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.074276 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.074850 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.074856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.075225 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:25.075701 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.075714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.076634 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.076638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.077814 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.077825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.078420 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.078425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.078628 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.079337 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:25.079354 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:25.080985 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:25.082841 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:25.082847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:25.082863 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:25.083135 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:25.083436 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:25.083440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:25.083442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:25.083592 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:25.084011 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:25.084018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:25.084327 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.084528 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.084879 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:12:25.087314 23287 heartbeater.cc:347] Connected to a master server at 127.18.90.62:42129
I20260806 16:12:25.087564 23287 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:25.088049 23287 heartbeater.cc:511] Master 127.18.90.62:42129 requested a full tablet report, sending...
I20260806 16:12:25.089286 22959 ts_manager.cc:194] Registered new tserver with Master: f3c36a637d24486898e2acff23ad261f (127.18.90.2:38169)
I20260806 16:12:25.089887 22959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:37229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:25.107333 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42129
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:25.220058 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:25.220459 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:25.220531 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:25.224661 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:25.224723 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:25.224783 23292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:25.230162 23292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:42129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:25.231194 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:25.232172 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:12:25.241881 23292 server_base.cc:1034] running on GCE node
W20260806 16:12:25.241829 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:25.241838 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:25.241827 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:25.242663 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:25.243355 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:25.244549 23292 hybrid_clock.cc:648] HybridClock initialized: now 1786032745244531 us; error 34 us; skew 500 ppm
Aug 06 16:12:25 dist-test-slave-0b58 krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:25.247438 23292 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:25.248229 23292 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:25.248363 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:25.248601 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:25.250674 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "f63c578f53c648ab924685c39b1a28b1"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "0a0c6d55fdcc94fe114715fef6f62139"
server_key_iv: "748d57f79c8e61d93e8cef7bafebb271"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.251257 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "f63c578f53c648ab924685c39b1a28b1"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "0a0c6d55fdcc94fe114715fef6f62139"
server_key_iv: "748d57f79c8e61d93e8cef7bafebb271"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.254992 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 16:12:25.257489 23306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.258757 23292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:25.258908 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "f63c578f53c648ab924685c39b1a28b1"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "0a0c6d55fdcc94fe114715fef6f62139"
server_key_iv: "748d57f79c8e61d93e8cef7bafebb271"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.259027 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:25.291803 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:25.295230 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:25.295462 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:25.296118 23292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:25.297058 23292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:25.297132 23292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.297202 23292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:25.297250 23292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.306473 23292 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:45501
I20260806 16:12:25.306504 23420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:45501 every 8 connection(s)
I20260806 16:12:25.307104 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:25.314481 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23292
I20260806 16:12:25.314643 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:25.314970 18792 external_mini_cluster.cc:1428] Setting key 2026477fd7e6bed43b6d3fd4dcdc0b13
Aug 06 16:12:25 dist-test-slave-0b58 krb5kdc[22914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:25.321413 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.308968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:51901 (local address 127.18.90.62:42129)
0806 16:12:25.309146 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.309151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.310366 (+ 1215us) server_negotiation.cc:407] Connection header received
0806 16:12:25.311342 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.311345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.311394 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.311489 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:25.312894 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.313441 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.314625 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.314914 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.315028 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.317911 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:25.317914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:25.317916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:25.317942 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:25.319311 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:25.319851 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:25.319854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:25.319855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:25.319938 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:25.320358 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:25.320361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:25.320362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:25.320553 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:25.320643 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.321170 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.321288 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:12:25.322257 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.309422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42129 (local address 127.18.90.3:51901)
0806 16:12:25.310207 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.310249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.311025 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.311683 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.311690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.312081 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:25.312728 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.312745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.313555 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.313557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.314487 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.314497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.315092 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.315097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.315277 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.315952 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:25.315977 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:25.317754 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:25.319427 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:25.319432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:25.319444 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:25.319748 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:25.320025 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:25.320027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:25.320028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:25.320178 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:25.320670 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:25.320683 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:25.320937 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.321106 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.321392 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260806 16:12:25.323427 23421 heartbeater.cc:347] Connected to a master server at 127.18.90.62:42129
I20260806 16:12:25.323657 23421 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:25.324077 23421 heartbeater.cc:511] Master 127.18.90.62:42129 requested a full tablet report, sending...
I20260806 16:12:25.325191 22959 ts_manager.cc:194] Registered new tserver with Master: f63c578f53c648ab924685c39b1a28b1 (127.18.90.3:45501)
I20260806 16:12:25.325793 22959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:51901
I20260806 16:12:25.329159 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:12:25.338637 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.331038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.18.90.62:42129)
0806 16:12:25.331185 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.331189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.331377 (+ 188us) server_negotiation.cc:407] Connection header received
0806 16:12:25.331527 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.331533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.331578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.331660 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:25.332409 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.332925 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.333784 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.333997 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.334105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.335511 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:25.335515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:25.335517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:25.335540 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:25.336889 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:25.337317 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:25.337322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:25.337324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:25.337406 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:25.337702 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:25.337707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:25.337709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:25.337942 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:25.338091 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.338350 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.338470 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:12:25.341706 22959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:25.344853 22959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:25.359428 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.353624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35757 (local address 127.0.0.1:53684)
0806 16:12:25.354012 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.354025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.354122 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.355468 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.355471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.355490 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:25.355604 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.355611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.356732 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.356737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.358019 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.358030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.358841 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.358844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.358963 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.359003 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.359113 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.359247 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":101,"thread_start_us":110,"threads_started":1}
I20260806 16:12:25.359572 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.354339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.18.90.1:35757)
0806 16:12:25.354926 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.354933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.354955 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:12:25.355030 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.355035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.355222 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.355501 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:25.355728 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.356613 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.358171 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.358704 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.358837 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.358991 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.359330 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.359431 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":100,"threads_started":1}
I20260806 16:12:25.359992 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.353991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:45501 (local address 127.0.0.1:39164)
0806 16:12:25.354286 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.354299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.354386 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.355119 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.355122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.355140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:25.355234 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.355240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.356677 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.356681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.358019 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.358030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.359101 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.359107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.359247 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.359267 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.359337 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.359392 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":87,"thread_start_us":93,"threads_started":1}
I20260806 16:12:25.359990 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.353239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38169 (local address 127.0.0.1:59748)
0806 16:12:25.353612 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.353665 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.353807 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.354640 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.354646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.354671 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:25.354802 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.354809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.356328 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.356332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.357553 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.357565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.359026 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.359035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.359188 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.359211 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.359283 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.359367 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":79,"thread_start_us":105,"threads_started":1}
I20260806 16:12:25.360435 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.353425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.18.90.2:38169)
0806 16:12:25.353865 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.353880 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.353909 (+ 29us) server_negotiation.cc:407] Connection header received
0806 16:12:25.353992 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.353998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.354253 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.354423 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:25.354929 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.356155 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.357690 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.358521 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.358622 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.358769 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.359498 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.359594 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":140,"threads_started":1}
I20260806 16:12:25.360598 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.354351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.18.90.3:45501)
0806 16:12:25.354655 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.354667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.354687 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:12:25.354773 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.354778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.354974 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.355375 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:25.355524 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.356544 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.358165 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.358989 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.359099 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.359237 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.360336 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.360443 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260806 16:12:25.362617 23084 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7f8524a6204e0c99890c74f1142ac1 (DEFAULT_TABLE table=test-table [id=13f69d93f23f4c4caef0523dfc8feddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:25.363560 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7f8524a6204e0c99890c74f1142ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:25.363165 23355 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7f8524a6204e0c99890c74f1142ac1 (DEFAULT_TABLE table=test-table [id=13f69d93f23f4c4caef0523dfc8feddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:25.363165 23220 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7f8524a6204e0c99890c74f1142ac1 (DEFAULT_TABLE table=test-table [id=13f69d93f23f4c4caef0523dfc8feddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:25.364184 23355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7f8524a6204e0c99890c74f1142ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:25.364179 23220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7f8524a6204e0c99890c74f1142ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:25.370730 23438 tablet_bootstrap.cc:492] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: Bootstrap starting.
I20260806 16:12:25.371068 23439 tablet_bootstrap.cc:492] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Bootstrap starting.
I20260806 16:12:25.371832 23440 tablet_bootstrap.cc:492] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Bootstrap starting.
I20260806 16:12:25.373257 23438 tablet_bootstrap.cc:654] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:25.373257 23439 tablet_bootstrap.cc:654] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:25.373579 23440 tablet_bootstrap.cc:654] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:25.374207 23438 log.cc:824] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:25.374348 23439 log.cc:824] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:25.374595 23440 log.cc:824] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:25.376345 23439 tablet_bootstrap.cc:492] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: No bootstrap required, opened a new log
I20260806 16:12:25.376345 23438 tablet_bootstrap.cc:492] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: No bootstrap required, opened a new log
I20260806 16:12:25.376540 23438 ts_tablet_manager.cc:1397] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:25.376540 23439 ts_tablet_manager.cc:1397] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:25.376684 23440 tablet_bootstrap.cc:492] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: No bootstrap required, opened a new log
I20260806 16:12:25.376892 23440 ts_tablet_manager.cc:1397] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:12:25.379966 23440 raft_consensus.cc:348] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.380213 23440 raft_consensus.cc:374] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:25.380293 23440 raft_consensus.cc:729] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63c578f53c648ab924685c39b1a28b1, State: Initialized, Role: FOLLOWER
I20260806 16:12:25.380734 23440 consensus_queue.cc:260] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.380709 23439 raft_consensus.cc:348] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.380975 23439 raft_consensus.cc:374] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:25.381038 23439 raft_consensus.cc:729] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c36a637d24486898e2acff23ad261f, State: Initialized, Role: FOLLOWER
I20260806 16:12:25.381016 23438 raft_consensus.cc:348] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.381246 23438 raft_consensus.cc:374] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:25.381317 23438 raft_consensus.cc:729] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedbc1179c54469c927ad969e42d6906, State: Initialized, Role: FOLLOWER
I20260806 16:12:25.381493 23439 consensus_queue.cc:260] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.381585 23421 heartbeater.cc:503] Master 127.18.90.62:42129 was elected leader, sending a full tablet report...
I20260806 16:12:25.381727 23438 consensus_queue.cc:260] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.382211 23440 ts_tablet_manager.cc:1428] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:12:25.382448 23439 ts_tablet_manager.cc:1428] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:25.382521 23438 ts_tablet_manager.cc:1428] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:12:25.382756 23287 heartbeater.cc:503] Master 127.18.90.62:42129 was elected leader, sending a full tablet report...
I20260806 16:12:25.382779 23151 heartbeater.cc:503] Master 127.18.90.62:42129 was elected leader, sending a full tablet report...
W20260806 16:12:25.558077 23422 tablet.cc:2369] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:12:25.571301 23288 tablet.cc:2369] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:25.582288 23446 raft_consensus.cc:482] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:25.582496 23446 raft_consensus.cc:504] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.583586 23446 leader_election.cc:288] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c36a637d24486898e2acff23ad261f (127.18.90.2:38169), f63c578f53c648ab924685c39b1a28b1 (127.18.90.3:45501)
I20260806 16:12:25.588316 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.584006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:60083 (local address 127.18.90.2:38169)
0806 16:12:25.584159 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.584164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.584410 (+ 246us) server_negotiation.cc:407] Connection header received
0806 16:12:25.584607 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.584610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.584676 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.584782 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:25.585137 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.585741 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.587089 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.587711 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.587828 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.587896 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.588065 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.588146 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:25.588344 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.583928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38169 (local address 127.18.90.1:60083)
0806 16:12:25.584247 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.584310 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.584461 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.584809 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.584812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.584839 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:25.584996 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.585003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.585856 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.585860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.586951 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.586966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.587850 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.587853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.587930 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.587952 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.588046 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.588164 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":24,"spinlock_wait_cycles":3328,"thread_start_us":142,"threads_started":1}
I20260806 16:12:25.588572 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.584573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:36541 (local address 127.18.90.3:45501)
0806 16:12:25.584731 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.584754 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.584857 (+ 103us) server_negotiation.cc:407] Connection header received
0806 16:12:25.584969 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.584972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.585027 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.585107 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:25.585401 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.585941 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.587315 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.587925 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.588022 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.588082 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.588340 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.588424 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 16:12:25.588685 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.584430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:45501 (local address 127.18.90.1:36541)
0806 16:12:25.584769 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.584783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:25.584877 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:25.585130 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:25.585133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:25.585150 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:25.585286 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.585292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.586054 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.586056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.587119 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:25.587129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.588026 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.588029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:25.588177 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.588195 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:12:25.588320 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 16:12:25.588532 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":21,"thread_start_us":101,"threads_started":1}
I20260806 16:12:25.589294 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7f8524a6204e0c99890c74f1142ac1" candidate_uuid: "cedbc1179c54469c927ad969e42d6906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63c578f53c648ab924685c39b1a28b1" is_pre_election: true
I20260806 16:12:25.589298 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7f8524a6204e0c99890c74f1142ac1" candidate_uuid: "cedbc1179c54469c927ad969e42d6906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c36a637d24486898e2acff23ad261f" is_pre_election: true
I20260806 16:12:25.589630 23375 raft_consensus.cc:2454] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedbc1179c54469c927ad969e42d6906 in term 0.
I20260806 16:12:25.589630 23241 raft_consensus.cc:2454] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cedbc1179c54469c927ad969e42d6906 in term 0.
I20260806 16:12:25.590121 23038 leader_election.cc:302] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cedbc1179c54469c927ad969e42d6906, f63c578f53c648ab924685c39b1a28b1; no voters:
I20260806 16:12:25.590368 23446 raft_consensus.cc:2781] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:25.590469 23446 raft_consensus.cc:482] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:25.590528 23446 raft_consensus.cc:3037] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:25.591583 23446 raft_consensus.cc:504] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.591945 23446 leader_election.cc:288] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [CANDIDATE]: Term 1 election: Requested vote from peers f3c36a637d24486898e2acff23ad261f (127.18.90.2:38169), f63c578f53c648ab924685c39b1a28b1 (127.18.90.3:45501)
I20260806 16:12:25.592350 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7f8524a6204e0c99890c74f1142ac1" candidate_uuid: "cedbc1179c54469c927ad969e42d6906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c36a637d24486898e2acff23ad261f"
I20260806 16:12:25.592412 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7f8524a6204e0c99890c74f1142ac1" candidate_uuid: "cedbc1179c54469c927ad969e42d6906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f63c578f53c648ab924685c39b1a28b1"
I20260806 16:12:25.592506 23241 raft_consensus.cc:3037] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:25.592537 23375 raft_consensus.cc:3037] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 0 FOLLOWER]: Advancing to term 1
W20260806 16:12:25.593365 23152 tablet.cc:2369] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:25.593659 23375 raft_consensus.cc:2454] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedbc1179c54469c927ad969e42d6906 in term 1.
I20260806 16:12:25.593737 23241 raft_consensus.cc:2454] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cedbc1179c54469c927ad969e42d6906 in term 1.
I20260806 16:12:25.594067 23038 leader_election.cc:302] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cedbc1179c54469c927ad969e42d6906, f3c36a637d24486898e2acff23ad261f; no voters:
I20260806 16:12:25.594218 23446 raft_consensus.cc:2781] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:25.594470 23446 raft_consensus.cc:686] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 1 LEADER]: Becoming Leader. State: Replica: cedbc1179c54469c927ad969e42d6906, State: Running, Role: LEADER
I20260806 16:12:25.594836 23446 consensus_queue.cc:237] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } }
I20260806 16:12:25.597908 22959 catalog_manager.cc:5495] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 reported cstate change: term changed from 0 to 1, leader changed from <none> to cedbc1179c54469c927ad969e42d6906 (127.18.90.1). New cstate: current_term: 1 leader_uuid: "cedbc1179c54469c927ad969e42d6906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cedbc1179c54469c927ad969e42d6906" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35757 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:25.653067 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.649449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.18.90.1:35757)
0806 16:12:25.649580 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.649583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.649596 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:25.649677 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.649679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.649741 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.649830 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:25.650056 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.650550 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.651356 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:25.651653 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:25.651788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:25.652172 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:25.652661 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:25.652809 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.652852 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.652905 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:12:25.658977 23375 raft_consensus.cc:1261] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 1 FOLLOWER]: Refusing update from remote peer cedbc1179c54469c927ad969e42d6906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:25.658977 23241 raft_consensus.cc:1261] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 1 FOLLOWER]: Refusing update from remote peer cedbc1179c54469c927ad969e42d6906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:25.659682 23446 consensus_queue.cc:1035] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f63c578f53c648ab924685c39b1a28b1" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:25.659868 23449 consensus_queue.cc:1035] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c36a637d24486898e2acff23ad261f" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:25.667536 23453 mvcc.cc:205] Tried to move back new op lower bound from 7315590126211940352 to 7315590125958840320. Current Snapshot: MvccSnapshot[applied={T|T < 7315590126211940352}]
I20260806 16:12:25.670028 23454 mvcc.cc:205] Tried to move back new op lower bound from 7315590126211940352 to 7315590125958840320. Current Snapshot: MvccSnapshot[applied={T|T < 7315590126211940352}]
I20260806 16:12:25.675496 22958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:25.675644 22958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:25.677983 22958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fad60320bf04854a4920c1f379f23cd: Sending DeleteTablet for 3 replicas of tablet 1f7f8524a6204e0c99890c74f1142ac1
I20260806 16:12:25.678862 23355 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7f8524a6204e0c99890c74f1142ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:25 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:39164
I20260806 16:12:25.678877 23084 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7f8524a6204e0c99890c74f1142ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:25 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:53684
I20260806 16:12:25.679340 23464 tablet_replica.cc:330] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906: stopping tablet replica
I20260806 16:12:25.679358 23220 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7f8524a6204e0c99890c74f1142ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:25 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:59748
I20260806 16:12:25.679574 23464 raft_consensus.cc:2229] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:25.679816 23465 tablet_replica.cc:330] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: stopping tablet replica
I20260806 16:12:25.679983 23464 raft_consensus.cc:2258] T 1f7f8524a6204e0c99890c74f1142ac1 P cedbc1179c54469c927ad969e42d6906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:25.680099 23465 raft_consensus.cc:2229] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:25.680438 23465 raft_consensus.cc:2258] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:25.680540 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23021
I20260806 16:12:25.681342 23463 tablet_replica.cc:330] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: stopping tablet replica
I20260806 16:12:25.681605 23463 raft_consensus.cc:2229] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:25.681888 23463 raft_consensus.cc:2258] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:25.682065 23465 ts_tablet_manager.cc:1905] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:25.683506 23463 ts_tablet_manager.cc:1905] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:25.685249 23465 ts_tablet_manager.cc:1918] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:25.685371 23465 log.cc:1196] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/1f7f8524a6204e0c99890c74f1142ac1
I20260806 16:12:25.685693 23465 ts_tablet_manager.cc:1939] T 1f7f8524a6204e0c99890c74f1142ac1 P f3c36a637d24486898e2acff23ad261f: Deleting consensus metadata
I20260806 16:12:25.686376 23463 ts_tablet_manager.cc:1918] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:25.686476 23463 log.cc:1196] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/1f7f8524a6204e0c99890c74f1142ac1
I20260806 16:12:25.686831 23463 ts_tablet_manager.cc:1939] T 1f7f8524a6204e0c99890c74f1142ac1 P f63c578f53c648ab924685c39b1a28b1: Deleting consensus metadata
I20260806 16:12:25.687042 22945 catalog_manager.cc:4841] TS f3c36a637d24486898e2acff23ad261f (127.18.90.2:38169): tablet 1f7f8524a6204e0c99890c74f1142ac1 (table test-table [id=13f69d93f23f4c4caef0523dfc8feddc]) successfully deleted
I20260806 16:12:25.687914 22945 catalog_manager.cc:4841] TS f63c578f53c648ab924685c39b1a28b1 (127.18.90.3:45501): tablet 1f7f8524a6204e0c99890c74f1142ac1 (table test-table [id=13f69d93f23f4c4caef0523dfc8feddc]) successfully deleted
I20260806 16:12:25.690215 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23156
W20260806 16:12:25.690292 22943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:35757 (error 108)
I20260806 16:12:25.693683 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.693378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35757 (local address 127.0.0.1:53688)
0806 16:12:25.693521 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:25.693596 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:35757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 16:12:25.693959 22943 catalog_manager.cc:4584] TS cedbc1179c54469c927ad969e42d6906 (127.18.90.1:35757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f7f8524a6204e0c99890c74f1142ac1: Network error: Client connection negotiation failed: client connection to 127.18.90.1:35757: connect: Connection refused (error 111)
I20260806 16:12:25.697940 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23292
I20260806 16:12:25.704530 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 22927
2026-08-06T16:12:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2515 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T16:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:25Z Disabled control of system clock
I20260806 16:12:25.737112 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38145
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:38145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 16:12:25.851393 23470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:25.851691 23470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:25.851728 23470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:25.855898 23470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:25.855973 23470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:25.855988 23470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:25.856005 23470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:25.856021 23470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:25.860945 23470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:38145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:25.862140 23470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:25.863636 23470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:25.872304 23475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:25.872402 23470 server_base.cc:1034] running on GCE node
W20260806 16:12:25.872210 23478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:25.872205 23476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:25.873252 23470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:25.874361 23470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:25.875528 23470 hybrid_clock.cc:648] HybridClock initialized: now 1786032745875501 us; error 41 us; skew 500 ppm
I20260806 16:12:25.877516 23470 webserver.cc:460] Webserver started at http://127.18.90.62:42221/ using document root <none> and password file <none>
I20260806 16:12:25.878140 23470 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:25.878209 23470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:25.878460 23470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:25.880692 23470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "2f9145553f43d55043cad8631c47d336"
server_key_iv: "352d509cf9a195ea16a581838e4fd8e0"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.881295 23470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "2f9145553f43d55043cad8631c47d336"
server_key_iv: "352d509cf9a195ea16a581838e4fd8e0"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.885257 23470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:25.887895 23484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:25.889250 23470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:25.889403 23470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22"
format_stamp: "Formatted at 2026-08-06 16:12:25 on dist-test-slave-0b58"
server_key: "2f9145553f43d55043cad8631c47d336"
server_key_iv: "352d509cf9a195ea16a581838e4fd8e0"
server_key_version: "encryptionkey@0"
I20260806 16:12:25.889532 23470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:25.894431 23470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:25.895100 23470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:25.895299 23470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:25.902519 23470 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:38145
I20260806 16:12:25.902551 23537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:38145 every 8 connection(s)
I20260806 16:12:25.903573 23470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:25.907029 23538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:25.912678 23538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22: Bootstrap starting.
I20260806 16:12:25.913295 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23470
I20260806 16:12:25.913440 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:25.913722 18792 external_mini_cluster.cc:1428] Setting key 05bb6f7f1569ff7a69e0f249366df91c
I20260806 16:12:25.915560 23538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:25.916504 23538 log.cc:824] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:25.918897 23538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22: No bootstrap required, opened a new log
I20260806 16:12:25.919760 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:25.915067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.18.90.62:38145)
0806 16:12:25.915825 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:25.915850 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:25.915897 (+ 47us) server_negotiation.cc:407] Connection header received
0806 16:12:25.916801 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:25.916827 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:25.917185 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:25.917553 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:25.918274 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:25.918282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:25.918296 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:25.918308 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:25.918375 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:25.918398 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:25.918517 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:25.918589 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 16:12:25.918837 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":128,"threads_started":1}
I20260806 16:12:25.923014 23538 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38145 } }
I20260806 16:12:25.923242 23538 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:25.923301 23538 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e76bdfcd3a41f28ef73b6eef4b4b22, State: Initialized, Role: FOLLOWER
I20260806 16:12:25.923769 23538 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38145 } }
I20260806 16:12:25.923959 23538 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:25.924031 23538 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:25.924124 23538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:25.925199 23538 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38145 } }
I20260806 16:12:25.925544 23538 leader_election.cc:302] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e76bdfcd3a41f28ef73b6eef4b4b22; no voters:
I20260806 16:12:25.925854 23538 leader_election.cc:288] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:25.925985 23543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:25.926257 23543 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [term 1 LEADER]: Becoming Leader. State: Replica: d4e76bdfcd3a41f28ef73b6eef4b4b22, State: Running, Role: LEADER
I20260806 16:12:25.926566 23543 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38145 } }
I20260806 16:12:25.927163 23538 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:25.928277 23545 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e76bdfcd3a41f28ef73b6eef4b4b22. Latest consensus state: current_term: 1 leader_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38145 } } }
I20260806 16:12:25.928427 23545 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:25.928666 23544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e76bdfcd3a41f28ef73b6eef4b4b22" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38145 } } }
I20260806 16:12:25.928757 23544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:25.928895 23552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:25.931967 23552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:25.937893 23552 catalog_manager.cc:1347] Generated new cluster ID: 155f90297c174b238af8843f40f46e72
I20260806 16:12:25.937965 23552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:25.959575 23552 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:25.959952 23552 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:25.977414 23552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22: Generated new TSK 0
I20260806 16:12:25.978338 23552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:12:25.985787 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38145
--builtin_ntp_servers=127.18.90.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 16:12:26.101424 23562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:26.101740 23562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:26.101832 23562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:26.105820 23562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:26.105901 23562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:26.105998 23562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:26.110854 23562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:26.112087 23562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:26.113577 23562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:26.123785 23568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.123811 23567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:26.123899 23562 server_base.cc:1034] running on GCE node
W20260806 16:12:26.123811 23570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:26.124696 23562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:26.125567 23562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:26.126861 23562 hybrid_clock.cc:648] HybridClock initialized: now 1786032746126841 us; error 31 us; skew 500 ppm
I20260806 16:12:26.129106 23562 webserver.cc:460] Webserver started at http://127.18.90.1:38133/ using document root <none> and password file <none>
I20260806 16:12:26.129786 23562 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:26.129858 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:26.130098 23562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:26.132326 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "615e720745bc49bcab8a52cf4391b7ce"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "ecfe8d6d3eb5dec73f0b88dd69b1a170"
server_key_iv: "3cc9fc43ad19bd8556ef4d9299e0934e"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.132929 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "615e720745bc49bcab8a52cf4391b7ce"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "ecfe8d6d3eb5dec73f0b88dd69b1a170"
server_key_iv: "3cc9fc43ad19bd8556ef4d9299e0934e"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.136735 23562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:26.139392 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.140694 23562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:26.140821 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "615e720745bc49bcab8a52cf4391b7ce"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "ecfe8d6d3eb5dec73f0b88dd69b1a170"
server_key_iv: "3cc9fc43ad19bd8556ef4d9299e0934e"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.140938 23562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:26.145824 23562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:26.146492 23562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:26.146687 23562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:26.147253 23562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:26.148196 23562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:26.148331 23562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.148415 23562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:26.148469 23562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.157537 23562 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:38993
I20260806 16:12:26.157560 23691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:38993 every 8 connection(s)
I20260806 16:12:26.158527 23562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:26.162928 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23562
I20260806 16:12:26.163079 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:26.163364 18792 external_mini_cluster.cc:1428] Setting key c6d4a747149ff4ed1521a2f7439b8b5a
I20260806 16:12:26.164949 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.160392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:59393 (local address 127.18.90.62:38145)
0806 16:12:26.160545 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.160549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.161683 (+ 1134us) server_negotiation.cc:407] Connection header received
0806 16:12:26.163014 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.163018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.163064 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.163146 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.164472 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.164475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.164477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.164489 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.164510 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.164518 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.164599 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.164770 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.164816 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:26.165748 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38145
--builtin_ntp_servers=127.18.90.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 16:12:26.165925 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.160838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38145 (local address 127.18.90.1:59393)
0806 16:12:26.161528 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.161572 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.162789 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.163327 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.163334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.163425 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.164129 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.164130 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.164187 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.164316 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.164613 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.164622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.164628 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.164811 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.165065 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260806 16:12:26.167294 23692 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38145
I20260806 16:12:26.167537 23692 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:26.167969 23692 heartbeater.cc:511] Master 127.18.90.62:38145 requested a full tablet report, sending...
I20260806 16:12:26.169616 23502 ts_manager.cc:194] Registered new tserver with Master: 615e720745bc49bcab8a52cf4391b7ce (127.18.90.1:38993)
W20260806 16:12:26.286494 23695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:26.286815 23695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:26.286887 23695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:26.291152 23695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:26.291270 23695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:26.291363 23695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:26.297003 23695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:26.298226 23695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:26.299672 23695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:12:26.309825 23695 server_base.cc:1034] running on GCE node
W20260806 16:12:26.309787 23700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.309755 23703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.309787 23701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:26.310527 23695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:26.311146 23695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:26.312364 23695 hybrid_clock.cc:648] HybridClock initialized: now 1786032746312338 us; error 37 us; skew 500 ppm
I20260806 16:12:26.314234 23695 webserver.cc:460] Webserver started at http://127.18.90.2:46243/ using document root <none> and password file <none>
I20260806 16:12:26.314857 23695 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:26.314953 23695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:26.315192 23695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:26.317646 23695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "5558566bfa1f406091bf0f0d97a7a91d"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "0b9c099fe766e0d5a162889f7d740ca3"
server_key_iv: "9a2abf95520eb943afaf58f416f17bbd"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.318333 23695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "5558566bfa1f406091bf0f0d97a7a91d"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "0b9c099fe766e0d5a162889f7d740ca3"
server_key_iv: "9a2abf95520eb943afaf58f416f17bbd"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.322321 23695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:26.325022 23709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.326406 23695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:26.326563 23695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "5558566bfa1f406091bf0f0d97a7a91d"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "0b9c099fe766e0d5a162889f7d740ca3"
server_key_iv: "9a2abf95520eb943afaf58f416f17bbd"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.326694 23695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:26.331743 23695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:26.332489 23695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:26.332683 23695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:26.333256 23695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:26.334200 23695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:26.334275 23695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.334364 23695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:26.334414 23695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.343176 23695 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:36235
I20260806 16:12:26.343232 23824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:36235 every 8 connection(s)
I20260806 16:12:26.344164 23695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:26.345996 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23695
I20260806 16:12:26.346217 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:26.346580 18792 external_mini_cluster.cc:1428] Setting key 21b623b5cd4ccaff8b48a2b5575e2689
I20260806 16:12:26.348971 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38145
--builtin_ntp_servers=127.18.90.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 16:12:26.351572 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.346283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:48591 (local address 127.18.90.62:38145)
0806 16:12:26.346452 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.346456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.347757 (+ 1301us) server_negotiation.cc:407] Connection header received
0806 16:12:26.349617 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.349620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.349656 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.349727 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.351038 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.351043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.351048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.351065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.351094 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.351106 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.351208 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.351345 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.351397 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:12:26.352533 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.346776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38145 (local address 127.18.90.2:48591)
0806 16:12:26.347602 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.347647 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.349373 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.349917 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.349929 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.350052 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.350659 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.350663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.350736 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.350847 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.351202 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.351210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.351215 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.351358 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.351559 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":167,"threads_started":1}
I20260806 16:12:26.353737 23825 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38145
I20260806 16:12:26.353971 23825 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:26.354398 23825 heartbeater.cc:511] Master 127.18.90.62:38145 requested a full tablet report, sending...
I20260806 16:12:26.355515 23502 ts_manager.cc:194] Registered new tserver with Master: 5558566bfa1f406091bf0f0d97a7a91d (127.18.90.2:36235)
W20260806 16:12:26.468588 23828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:26.468914 23828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:26.468992 23828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:26.474293 23828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:26.474382 23828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:26.474479 23828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:26.479688 23828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:26.480993 23828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:26.482432 23828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:12:26.492527 23828 server_base.cc:1034] running on GCE node
W20260806 16:12:26.492473 23834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.492489 23833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.492467 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:26.493242 23828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:26.493952 23828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:26.495111 23828 hybrid_clock.cc:648] HybridClock initialized: now 1786032746495074 us; error 48 us; skew 500 ppm
I20260806 16:12:26.497071 23828 webserver.cc:460] Webserver started at http://127.18.90.3:38755/ using document root <none> and password file <none>
I20260806 16:12:26.497673 23828 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:26.497752 23828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:26.498004 23828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:26.500089 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "65fc61ca952d4ac2828e8f74a2dfa05c"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "bac742f1090259432500c9891cd9e016"
server_key_iv: "8874471ff007ab6addab7f100dde66eb"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.500736 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "65fc61ca952d4ac2828e8f74a2dfa05c"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "bac742f1090259432500c9891cd9e016"
server_key_iv: "8874471ff007ab6addab7f100dde66eb"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.504555 23828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:26.507288 23843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.508611 23828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:26.508754 23828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "65fc61ca952d4ac2828e8f74a2dfa05c"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "bac742f1090259432500c9891cd9e016"
server_key_iv: "8874471ff007ab6addab7f100dde66eb"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.508878 23828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:26.513464 23828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:26.514076 23828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:26.514266 23828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:26.514837 23828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:26.515792 23828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:26.515870 23828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.515944 23828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:26.515997 23828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.524722 23828 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:43187
I20260806 16:12:26.524794 23957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:43187 every 8 connection(s)
I20260806 16:12:26.525672 23828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:26.526809 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 23828
I20260806 16:12:26.526932 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:26.527266 18792 external_mini_cluster.cc:1428] Setting key 90ed68db232873690f2ae3a336f3ca3c
I20260806 16:12:26.533139 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.529496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:51759 (local address 127.18.90.62:38145)
0806 16:12:26.529627 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.529655 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.530129 (+ 474us) server_negotiation.cc:407] Connection header received
0806 16:12:26.531563 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.531566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.531604 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.531678 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.532776 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.532779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.532781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.532791 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.532812 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.532821 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.532899 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.532995 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.533032 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:12:26.534044 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.528431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38145 (local address 127.18.90.3:51759)
0806 16:12:26.529938 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.530021 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.531363 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.531857 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.531864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.531952 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.532531 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.532533 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.532574 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.532640 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.532887 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.532891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.532894 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.533014 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.533237 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":131,"threads_started":1}
I20260806 16:12:26.535226 23958 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38145
I20260806 16:12:26.535470 23958 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:26.535890 23958 heartbeater.cc:511] Master 127.18.90.62:38145 requested a full tablet report, sending...
I20260806 16:12:26.537025 23502 ts_manager.cc:194] Registered new tserver with Master: 65fc61ca952d4ac2828e8f74a2dfa05c (127.18.90.3:43187)
I20260806 16:12:26.541532 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:12:26.545249 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.543524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.18.90.62:38145)
0806 16:12:26.543684 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.543689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.543821 (+ 132us) server_negotiation.cc:407] Connection header received
0806 16:12:26.543921 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.543923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.543956 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.544022 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.544818 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.544821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.544823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.544833 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.544851 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.544858 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.544929 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.545031 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.545064 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 16:12:26.548342 23502 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:26.551450 23502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:26.563566 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.560216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:43187 (local address 127.0.0.1:57782)
0806 16:12:26.560782 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.560833 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.560939 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.562154 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.562158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.562174 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.562761 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.562763 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.562784 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.562857 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.563212 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.563222 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.563230 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.563331 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.563404 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":89,"threads_started":1}
I20260806 16:12:26.563714 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.560416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:38993 (local address 127.0.0.1:35128)
0806 16:12:26.561132 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.561145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.561213 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.561969 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.561975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.562057 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.562986 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.562988 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.562996 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.563015 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.563406 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.563409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.563412 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.563493 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.563531 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":107,"threads_started":1}
I20260806 16:12:26.563750 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.560994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.18.90.1:38993)
0806 16:12:26.561469 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.561483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.561508 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:12:26.561572 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.561578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.561792 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.561953 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.563152 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.563161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.563175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.563190 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.563235 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.563269 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.563409 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.563504 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.563578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":136,"threads_started":1}
I20260806 16:12:26.564536 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.560988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36235 (local address 127.0.0.1:44258)
0806 16:12:26.561590 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.561603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.561671 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.562721 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.562727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.562743 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.563186 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.563196 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.563208 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.563239 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.564208 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.564212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.564215 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.564371 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.564411 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":104,"threads_started":1}
I20260806 16:12:26.564779 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.560406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.18.90.3:43187)
0806 16:12:26.561420 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.561432 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.561458 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:12:26.561807 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.561812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.562006 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.562371 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.562994 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.563000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.563014 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.563023 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.563067 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.563098 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.564361 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.564412 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.564482 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":168,"threads_started":1}
I20260806 16:12:26.564952 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.561422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.18.90.2:36235)
0806 16:12:26.561845 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.561859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.561882 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:12:26.561934 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.561939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.562106 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.562229 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.563650 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.563655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.563665 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.563675 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.563726 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.564105 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.564233 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.564710 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.564788 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":132,"threads_started":1}
I20260806 16:12:26.567317 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 44b326236ab84671b65c6a0228f57605 (DEFAULT_TABLE table=test-table [id=3f1b58d7b1fa45298a223d68655d9950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:26.567231 23626 tablet_service.cc:1467] Processing CreateTablet for tablet 44b326236ab84671b65c6a0228f57605 (DEFAULT_TABLE table=test-table [id=3f1b58d7b1fa45298a223d68655d9950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:26.567512 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 44b326236ab84671b65c6a0228f57605 (DEFAULT_TABLE table=test-table [id=3f1b58d7b1fa45298a223d68655d9950]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:26.568409 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b326236ab84671b65c6a0228f57605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:26.568411 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b326236ab84671b65c6a0228f57605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:26.568411 23626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b326236ab84671b65c6a0228f57605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:26.575286 23975 tablet_bootstrap.cc:492] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Bootstrap starting.
I20260806 16:12:26.576452 23976 tablet_bootstrap.cc:492] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: Bootstrap starting.
I20260806 16:12:26.576452 23977 tablet_bootstrap.cc:492] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Bootstrap starting.
I20260806 16:12:26.578788 23976 tablet_bootstrap.cc:654] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:26.578788 23977 tablet_bootstrap.cc:654] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:26.578972 23975 tablet_bootstrap.cc:654] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:26.579716 23975 log.cc:824] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:26.579746 23977 log.cc:824] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:26.579746 23976 log.cc:824] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:26.581674 23977 tablet_bootstrap.cc:492] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: No bootstrap required, opened a new log
I20260806 16:12:26.581676 23976 tablet_bootstrap.cc:492] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: No bootstrap required, opened a new log
I20260806 16:12:26.581899 23976 ts_tablet_manager.cc:1397] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:12:26.581900 23977 ts_tablet_manager.cc:1397] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:26.585202 23975 tablet_bootstrap.cc:492] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: No bootstrap required, opened a new log
I20260806 16:12:26.585417 23975 ts_tablet_manager.cc:1397] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260806 16:12:26.585840 23977 raft_consensus.cc:348] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.586099 23977 raft_consensus.cc:374] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:26.586172 23977 raft_consensus.cc:729] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5558566bfa1f406091bf0f0d97a7a91d, State: Initialized, Role: FOLLOWER
I20260806 16:12:26.586431 23976 raft_consensus.cc:348] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.586683 23976 raft_consensus.cc:374] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:26.586634 23977 consensus_queue.cc:260] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.586753 23976 raft_consensus.cc:729] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615e720745bc49bcab8a52cf4391b7ce, State: Initialized, Role: FOLLOWER
I20260806 16:12:26.587188 23976 consensus_queue.cc:260] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.587754 23977 ts_tablet_manager.cc:1428] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:26.587916 23692 heartbeater.cc:503] Master 127.18.90.62:38145 was elected leader, sending a full tablet report...
I20260806 16:12:26.588196 23976 ts_tablet_manager.cc:1428] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:12:26.588449 23825 heartbeater.cc:503] Master 127.18.90.62:38145 was elected leader, sending a full tablet report...
I20260806 16:12:26.589946 23975 raft_consensus.cc:348] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.590253 23975 raft_consensus.cc:374] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:26.590323 23975 raft_consensus.cc:729] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fc61ca952d4ac2828e8f74a2dfa05c, State: Initialized, Role: FOLLOWER
I20260806 16:12:26.590766 23975 consensus_queue.cc:260] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.591715 23958 heartbeater.cc:503] Master 127.18.90.62:38145 was elected leader, sending a full tablet report...
I20260806 16:12:26.591843 23975 ts_tablet_manager.cc:1428] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260806 16:12:26.595189 23826 tablet.cc:2369] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:12:26.659821 23693 tablet.cc:2369] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:26.713461 23984 raft_consensus.cc:482] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:26.713716 23984 raft_consensus.cc:504] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.714874 23984 leader_election.cc:288] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615e720745bc49bcab8a52cf4391b7ce (127.18.90.1:38993), 5558566bfa1f406091bf0f0d97a7a91d (127.18.90.2:36235)
I20260806 16:12:26.716971 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.715159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:38993 (local address 127.18.90.3:44149)
0806 16:12:26.715325 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.715343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.715446 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.715762 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.715766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.715788 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.716308 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.716310 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.716324 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.716355 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.716690 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.716694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.716704 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.716799 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.716837 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 16:12:26.717028 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.715290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:44149 (local address 127.18.90.1:38993)
0806 16:12:26.715456 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.715461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.715481 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:12:26.715548 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.715552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.715598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.715678 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.716518 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.716524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.716530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.716546 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.716572 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.716584 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.716676 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.716822 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.716867 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:12:26.717326 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.715581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:33005 (local address 127.18.90.2:36235)
0806 16:12:26.715739 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.715744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.715893 (+ 149us) server_negotiation.cc:407] Connection header received
0806 16:12:26.716003 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.716007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.716040 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.716136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.716848 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.716851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.716854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.716865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.716886 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.716895 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.716982 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.717129 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.717176 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 16:12:26.717635 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.715432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36235 (local address 127.18.90.3:33005)
0806 16:12:26.715808 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.715819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:26.715923 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:26.716155 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:26.716158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:26.716172 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:26.716690 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:26.716692 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:26.716703 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:26.716727 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:26.716987 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:26.716992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:26.716996 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:12:26.717475 (+ 479us) client_negotiation.cc:240] Negotiation successful
0806 16:12:26.717516 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260806 16:12:26.717772 23646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b326236ab84671b65c6a0228f57605" candidate_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615e720745bc49bcab8a52cf4391b7ce" is_pre_election: true
I20260806 16:12:26.718096 23646 raft_consensus.cc:2454] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fc61ca952d4ac2828e8f74a2dfa05c in term 0.
I20260806 16:12:26.718248 23778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b326236ab84671b65c6a0228f57605" candidate_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5558566bfa1f406091bf0f0d97a7a91d" is_pre_election: true
I20260806 16:12:26.718530 23778 raft_consensus.cc:2454] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fc61ca952d4ac2828e8f74a2dfa05c in term 0.
I20260806 16:12:26.718532 23845 leader_election.cc:302] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615e720745bc49bcab8a52cf4391b7ce, 65fc61ca952d4ac2828e8f74a2dfa05c; no voters:
I20260806 16:12:26.718798 23984 raft_consensus.cc:2781] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:26.718890 23984 raft_consensus.cc:482] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:26.718945 23984 raft_consensus.cc:3037] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:26.719978 23984 raft_consensus.cc:504] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.720350 23984 leader_election.cc:288] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [CANDIDATE]: Term 1 election: Requested vote from peers 615e720745bc49bcab8a52cf4391b7ce (127.18.90.1:38993), 5558566bfa1f406091bf0f0d97a7a91d (127.18.90.2:36235)
I20260806 16:12:26.720602 23646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b326236ab84671b65c6a0228f57605" candidate_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615e720745bc49bcab8a52cf4391b7ce"
I20260806 16:12:26.720665 23778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b326236ab84671b65c6a0228f57605" candidate_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5558566bfa1f406091bf0f0d97a7a91d"
I20260806 16:12:26.720757 23646 raft_consensus.cc:3037] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:26.720809 23778 raft_consensus.cc:3037] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:26.721782 23646 raft_consensus.cc:2454] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fc61ca952d4ac2828e8f74a2dfa05c in term 1.
I20260806 16:12:26.721911 23778 raft_consensus.cc:2454] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fc61ca952d4ac2828e8f74a2dfa05c in term 1.
I20260806 16:12:26.722072 23845 leader_election.cc:302] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615e720745bc49bcab8a52cf4391b7ce, 65fc61ca952d4ac2828e8f74a2dfa05c; no voters:
I20260806 16:12:26.722241 23984 raft_consensus.cc:2781] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:26.722450 23984 raft_consensus.cc:686] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 1 LEADER]: Becoming Leader. State: Replica: 65fc61ca952d4ac2828e8f74a2dfa05c, State: Running, Role: LEADER
I20260806 16:12:26.722824 23984 consensus_queue.cc:237] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } }
I20260806 16:12:26.726066 23500 catalog_manager.cc:5495] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65fc61ca952d4ac2828e8f74a2dfa05c (127.18.90.3). New cstate: current_term: 1 leader_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fc61ca952d4ac2828e8f74a2dfa05c" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 } health_report { overall_health: UNKNOWN } } }
I20260806 16:12:26.757225 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.755605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.18.90.3:43187)
0806 16:12:26.755770 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:26.755774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:26.755790 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:12:26.755916 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:26.755919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:26.755983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:26.756082 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:26.756802 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:26.756806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:26.756808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:26.756820 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:26.756848 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:26.756856 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:26.756951 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:26.757033 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 16:12:26.757077 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 16:12:26.763437 23646 raft_consensus.cc:1261] T 44b326236ab84671b65c6a0228f57605 P 615e720745bc49bcab8a52cf4391b7ce [term 1 FOLLOWER]: Refusing update from remote peer 65fc61ca952d4ac2828e8f74a2dfa05c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:26.763461 23778 raft_consensus.cc:1261] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 1 FOLLOWER]: Refusing update from remote peer 65fc61ca952d4ac2828e8f74a2dfa05c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:26.764292 23986 consensus_queue.cc:1035] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5558566bfa1f406091bf0f0d97a7a91d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:26.764580 23984 consensus_queue.cc:1035] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [LEADER]: Connected to new peer: Peer: permanent_uuid: "615e720745bc49bcab8a52cf4391b7ce" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:26.772903 23991 mvcc.cc:205] Tried to move back new op lower bound from 7315590130734714880 to 7315590130579390464. Current Snapshot: MvccSnapshot[applied={T|T < 7315590130734714880}]
I20260806 16:12:26.780506 23502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:26.780710 23502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:26.783190 23502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e76bdfcd3a41f28ef73b6eef4b4b22: Sending DeleteTablet for 3 replicas of tablet 44b326236ab84671b65c6a0228f57605
I20260806 16:12:26.784102 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b326236ab84671b65c6a0228f57605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:26 UTC) from {username='slave'} at 127.0.0.1:57782
I20260806 16:12:26.784430 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b326236ab84671b65c6a0228f57605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:26 UTC) from {username='slave'} at 127.0.0.1:44258
I20260806 16:12:26.784703 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23562
I20260806 16:12:26.785478 23999 tablet_replica.cc:330] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: stopping tablet replica
I20260806 16:12:26.785833 23999 raft_consensus.cc:2229] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:26.786270 23999 raft_consensus.cc:2258] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:26.786465 24000 tablet_replica.cc:330] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: stopping tablet replica
I20260806 16:12:26.786818 24000 raft_consensus.cc:2229] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:26.787113 24000 raft_consensus.cc:2258] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:26.788190 23999 ts_tablet_manager.cc:1905] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:26.788667 24000 ts_tablet_manager.cc:1905] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:26.791146 23999 ts_tablet_manager.cc:1918] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:26.791270 23999 log.cc:1196] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/44b326236ab84671b65c6a0228f57605
I20260806 16:12:26.791572 24000 ts_tablet_manager.cc:1918] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:26.791640 23999 ts_tablet_manager.cc:1939] T 44b326236ab84671b65c6a0228f57605 P 65fc61ca952d4ac2828e8f74a2dfa05c: Deleting consensus metadata
I20260806 16:12:26.791798 24000 log.cc:1196] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/44b326236ab84671b65c6a0228f57605
I20260806 16:12:26.792163 24000 ts_tablet_manager.cc:1939] T 44b326236ab84671b65c6a0228f57605 P 5558566bfa1f406091bf0f0d97a7a91d: Deleting consensus metadata
I20260806 16:12:26.793033 23488 catalog_manager.cc:4841] TS 65fc61ca952d4ac2828e8f74a2dfa05c (127.18.90.3:43187): tablet 44b326236ab84671b65c6a0228f57605 (table test-table [id=3f1b58d7b1fa45298a223d68655d9950]) successfully deleted
I20260806 16:12:26.793502 23486 catalog_manager.cc:4841] TS 5558566bfa1f406091bf0f0d97a7a91d (127.18.90.2:36235): tablet 44b326236ab84671b65c6a0228f57605 (table test-table [id=3f1b58d7b1fa45298a223d68655d9950]) successfully deleted
W20260806 16:12:26.793821 23486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:38993 (error 108)
I20260806 16:12:26.793936 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23695
I20260806 16:12:26.796579 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:26.796311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:38993 (local address 127.0.0.1:35130)
0806 16:12:26.796432 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:26.796506 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 16:12:26.796820 23486 catalog_manager.cc:4584] TS 615e720745bc49bcab8a52cf4391b7ce (127.18.90.1:38993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44b326236ab84671b65c6a0228f57605: Network error: Client connection negotiation failed: client connection to 127.18.90.1:38993: connect: Connection refused (error 111)
I20260806 16:12:26.802481 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23828
I20260806 16:12:26.808362 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 23470
2026-08-06T16:12:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1103 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 16:12:26.831030 18792 mini_oidc.cc:152] Starting JWKS server
I20260806 16:12:26.832930 18792 webserver.cc:460] Webserver started at https://127.0.0.1:37925/ using document root <none> and password file <none>
I20260806 16:12:26.833125 18792 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 16:12:26.833418 18792 webserver.cc:460] Webserver started at http://127.0.0.1:43523/ using document root <none> and password file <none>
2026-08-06T16:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:26Z Disabled control of system clock
I20260806 16:12:26.843492 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:40375
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:40375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37925/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/testchainca.cert with env {}
W20260806 16:12:26.956542 24007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:26.957124 24007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:26.957187 24007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:26.961448 24007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:26.961512 24007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:26.961527 24007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:26.961544 24007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:26.961560 24007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:12:26.961576 24007 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:12:26.961589 24007 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37925/jwks.json
I20260806 16:12:26.966995 24007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:40375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:40375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37925/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:26.968024 24007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:26.969105 24007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:26.978477 24015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.978482 24013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:26.978482 24012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:26.978780 24007 server_base.cc:1034] running on GCE node
I20260806 16:12:26.979418 24007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:26.980671 24007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:26.981839 24007 hybrid_clock.cc:648] HybridClock initialized: now 1786032746981827 us; error 30 us; skew 500 ppm
I20260806 16:12:26.983700 24007 webserver.cc:460] Webserver started at http://127.18.90.62:36895/ using document root <none> and password file <none>
I20260806 16:12:26.984360 24007 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:26.984428 24007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:26.984658 24007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:26.986779 24007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "2451717656474882932c77c16dd3c169"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "1636890c47648b0dcce53816da6e18cb"
server_key_iv: "ae4145de58f083292486c77197bdbf80"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.987363 24007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "2451717656474882932c77c16dd3c169"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "1636890c47648b0dcce53816da6e18cb"
server_key_iv: "ae4145de58f083292486c77197bdbf80"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.991143 24007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:26.993595 24021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:26.994834 24007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:26.994977 24007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "2451717656474882932c77c16dd3c169"
format_stamp: "Formatted at 2026-08-06 16:12:26 on dist-test-slave-0b58"
server_key: "1636890c47648b0dcce53816da6e18cb"
server_key_iv: "ae4145de58f083292486c77197bdbf80"
server_key_version: "encryptionkey@0"
I20260806 16:12:26.995095 24007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:27.028927 24007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:27.029716 24007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:27.029943 24007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:27.037483 24007 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:40375
I20260806 16:12:27.037519 24073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:40375 every 8 connection(s)
I20260806 16:12:27.038462 24007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:27.039784 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24007
I20260806 16:12:27.039921 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:27.040207 18792 external_mini_cluster.cc:1428] Setting key 3c1ca3266d4ea127e6cf123cf04432e1
I20260806 16:12:27.042157 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:27.048733 24074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169: Bootstrap starting.
I20260806 16:12:27.049556 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:27.041570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.18.90.62:40375)
0806 16:12:27.042171 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:27.042187 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:27.042250 (+ 63us) server_negotiation.cc:407] Connection header received
0806 16:12:27.042931 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:27.042948 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:27.043339 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:27.043614 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:27.043991 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.044971 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.046190 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.046650 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.046784 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.047708 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:27.047713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:27.047725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:27.048119 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:27.048173 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:27.048193 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:27.048338 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:27.048482 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 16:12:27.048788 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
I20260806 16:12:27.051224 24074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:27.052150 24074 log.cc:824] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:27.052639 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:12:27.054436 24074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169: No bootstrap required, opened a new log
I20260806 16:12:27.058152 24074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2451717656474882932c77c16dd3c169" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40375 } }
I20260806 16:12:27.058382 24074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:27.058465 24074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2451717656474882932c77c16dd3c169, State: Initialized, Role: FOLLOWER
I20260806 16:12:27.058914 24074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2451717656474882932c77c16dd3c169" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40375 } }
I20260806 16:12:27.059062 24074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:27.059126 24074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:27.059222 24074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:27.060139 24074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2451717656474882932c77c16dd3c169" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40375 } }
I20260806 16:12:27.060513 24074 leader_election.cc:302] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2451717656474882932c77c16dd3c169; no voters:
I20260806 16:12:27.060832 24074 leader_election.cc:288] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:27.060981 24079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:27.061218 24079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [term 1 LEADER]: Becoming Leader. State: Replica: 2451717656474882932c77c16dd3c169, State: Running, Role: LEADER
I20260806 16:12:27.061530 24079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2451717656474882932c77c16dd3c169" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40375 } }
I20260806 16:12:27.062062 24074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:27.063046 24080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2451717656474882932c77c16dd3c169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2451717656474882932c77c16dd3c169" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40375 } } }
I20260806 16:12:27.063177 24080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:27.063501 24081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2451717656474882932c77c16dd3c169. Latest consensus state: current_term: 1 leader_uuid: "2451717656474882932c77c16dd3c169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2451717656474882932c77c16dd3c169" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40375 } } }
I20260806 16:12:27.063616 24081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:27.063943 24088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:27.066751 24088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:27.072341 24088 catalog_manager.cc:1347] Generated new cluster ID: 9cc6a52f12a94cf9a371be2ab0fc6f2e
I20260806 16:12:27.072407 24088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:27.093459 24088 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:27.094399 24088 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:27.103968 24088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2451717656474882932c77c16dd3c169: Generated new TSK 0
I20260806 16:12:27.104701 24088 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 16:12:27.135643 24104 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 16:12:27.135870 24104 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 16:12:27.136543 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:27.114733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.18.90.62:40375)
0806 16:12:27.114855 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:27.114860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:27.115074 (+ 214us) server_negotiation.cc:407] Connection header received
0806 16:12:27.115328 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:27.115330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:27.115378 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:27.115483 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:12:27.115750 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.116328 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.117193 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.117410 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.117512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.117869 (+ 357us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:12:27.136044 (+ 18175us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 16:12:27.136396 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 16:12:27.136617 24077 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 16:12:27.137881 24103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:27.114653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:40375 (local address 127.0.0.1:56954)
0806 16:12:27.114990 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:27.115002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:27.115152 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:27.115500 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:27.115503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:27.115511 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:12:27.115642 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:27.115647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.116437 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.116441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:27.117071 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:27.117079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.117532 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.117536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:27.117635 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.117677 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:12:27.136313 (+ 18636us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 16:12:27.136570 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.90.62:40375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.90.62:40375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 16:12:27.138623 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24007
2026-08-06T16:12:27Z chronyd exiting
I20260806 16:12:27.177595 18792 test_util.cc:175] -----------------------------------------------
I20260806 16:12:27.177727 18792 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032720994684-18792-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 16:12:27.178938 18792 mini_oidc.cc:152] Starting JWKS server
I20260806 16:12:27.179828 18792 webserver.cc:460] Webserver started at https://127.0.0.1:39049/ using document root <none> and password file <none>
I20260806 16:12:27.179965 18792 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 16:12:27.180301 18792 webserver.cc:460] Webserver started at http://127.0.0.1:38593/ using document root <none> and password file <none>
2026-08-06T16:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:27Z Disabled control of system clock
I20260806 16:12:27.189905 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45923
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:45923
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39049/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/testchainca.cert with env {}
W20260806 16:12:27.301832 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:27.302115 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:27.302155 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:27.306104 24110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:27.306180 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:27.306195 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:27.306212 24110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:27.306227 24110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:12:27.306243 24110 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:12:27.306257 24110 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39049/jwks.json
I20260806 16:12:27.311779 24110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:45923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39049/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:27.312860 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:27.313861 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:27.322432 24115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:27.322454 24118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:27.322429 24116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:27.322767 24110 server_base.cc:1034] running on GCE node
I20260806 16:12:27.323398 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:27.324600 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:27.325768 24110 hybrid_clock.cc:648] HybridClock initialized: now 1786032747325736 us; error 48 us; skew 500 ppm
I20260806 16:12:27.327627 24110 webserver.cc:460] Webserver started at http://127.18.90.62:43351/ using document root <none> and password file <none>
I20260806 16:12:27.328223 24110 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:27.328346 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:27.328603 24110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:27.330691 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "8bd4996fc64a461da1b1fbf66fb1519b"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "f18801c399c2657196775ac5a1615c19"
server_key_iv: "be8b84ed3114ecf17edcaf37182188e8"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.331285 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "8bd4996fc64a461da1b1fbf66fb1519b"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "f18801c399c2657196775ac5a1615c19"
server_key_iv: "be8b84ed3114ecf17edcaf37182188e8"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.335029 24110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 16:12:27.337510 24124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:27.338820 24110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:27.338989 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "8bd4996fc64a461da1b1fbf66fb1519b"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "f18801c399c2657196775ac5a1615c19"
server_key_iv: "be8b84ed3114ecf17edcaf37182188e8"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.339112 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:27.359669 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:27.360471 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:27.360687 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:27.367880 24110 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:45923
I20260806 16:12:27.367895 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:45923 every 8 connection(s)
I20260806 16:12:27.368916 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:27.372143 24178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:27.376223 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24110
I20260806 16:12:27.376407 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:27.376708 18792 external_mini_cluster.cc:1428] Setting key dba22be9b3e84f5bbc5d70ef8b4b7633
I20260806 16:12:27.378168 24178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b: Bootstrap starting.
I20260806 16:12:27.380654 24178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:27.381414 24178 log.cc:824] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:27.383330 24178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b: No bootstrap required, opened a new log
I20260806 16:12:27.386519 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:27.378114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.18.90.62:45923)
0806 16:12:27.378801 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:27.378817 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:27.378855 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:12:27.379497 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:27.379515 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:27.379980 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:27.380379 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:27.380908 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.381946 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.382855 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.383394 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.383624 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.384599 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:27.384608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:27.384628 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:27.385176 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:27.385257 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:27.385289 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:27.385392 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:27.385515 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 16:12:27.385791 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
I20260806 16:12:27.387784 24178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45923 } }
I20260806 16:12:27.388059 24178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:27.388106 24178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd4996fc64a461da1b1fbf66fb1519b, State: Initialized, Role: FOLLOWER
I20260806 16:12:27.388610 24178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45923 } }
I20260806 16:12:27.388778 24178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:27.388854 24178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:27.388981 24178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:27.389598 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:12:27.390048 24178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45923 } }
I20260806 16:12:27.390386 24178 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd4996fc64a461da1b1fbf66fb1519b; no voters:
I20260806 16:12:27.390724 24178 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:27.390798 24183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:27.391017 24183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [term 1 LEADER]: Becoming Leader. State: Replica: 8bd4996fc64a461da1b1fbf66fb1519b, State: Running, Role: LEADER
I20260806 16:12:27.391377 24183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45923 } }
I20260806 16:12:27.392057 24178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:27.393112 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45923 } } }
I20260806 16:12:27.393271 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:27.393664 24192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:27.393635 24185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd4996fc64a461da1b1fbf66fb1519b. Latest consensus state: current_term: 1 leader_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd4996fc64a461da1b1fbf66fb1519b" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45923 } } }
I20260806 16:12:27.393797 24185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:27.396965 24192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:27.402949 24192 catalog_manager.cc:1347] Generated new cluster ID: 28af79e8774f4ba5b0b1a9891e27ed70
I20260806 16:12:27.403039 24192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:27.421732 24192 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:27.422812 24192 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:27.429090 24192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd4996fc64a461da1b1fbf66fb1519b: Generated new TSK 0
I20260806 16:12:27.429862 24192 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 16:12:27.464974 24208 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 16:12:27.465082 24208 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 16:12:27.465806 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:27.441878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.18.90.62:45923)
0806 16:12:27.442007 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:27.442012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:27.442227 (+ 215us) server_negotiation.cc:407] Connection header received
0806 16:12:27.442445 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:27.442448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:27.442497 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:27.442617 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:12:27.442929 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.443521 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.444720 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.444960 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.445097 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.445450 (+ 353us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:12:27.465417 (+ 19967us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 16:12:27.465674 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46046: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 16:12:27.465903 24181 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46046: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 16:12:27.466085 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:27.441778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45923 (local address 127.0.0.1:46046)
0806 16:12:27.442113 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:27.442125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:27.442283 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:27.442617 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:27.442621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:27.442630 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:12:27.442748 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:27.442755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.443643 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.443649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:27.444583 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:27.444593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.445127 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.445132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:27.445218 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.445241 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:12:27.465668 (+ 20427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 16:12:27.465888 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.90.62:45923: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
I20260806 16:12:27.466766 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24110
2026-08-06T16:12:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 16:12:27.507454 18792 mini_oidc.cc:152] Starting JWKS server
I20260806 16:12:27.508383 18792 webserver.cc:460] Webserver started at https://127.0.0.1:42127/ using document root <none> and password file <none>
I20260806 16:12:27.508472 18792 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 16:12:27.508699 18792 webserver.cc:460] Webserver started at http://127.0.0.1:35943/ using document root <none> and password file <none>
2026-08-06T16:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:27Z Disabled control of system clock
I20260806 16:12:27.519143 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45043
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:45043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42127/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 16:12:27.632046 24214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:27.632406 24214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:27.632452 24214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:27.636394 24214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:27.636447 24214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:27.636462 24214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:27.636479 24214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:27.636494 24214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:12:27.636510 24214 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:12:27.636524 24214 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42127/jwks.json
I20260806 16:12:27.641273 24214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:45043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42127/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:27.642360 24214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:27.643728 24214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:27.652340 24219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:27.652496 24214 server_base.cc:1034] running on GCE node
W20260806 16:12:27.652382 24222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:27.652382 24220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:27.653259 24214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:27.654403 24214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:27.655601 24214 hybrid_clock.cc:648] HybridClock initialized: now 1786032747655566 us; error 41 us; skew 500 ppm
I20260806 16:12:27.657601 24214 webserver.cc:460] Webserver started at http://127.18.90.62:42185/ using document root <none> and password file <none>
I20260806 16:12:27.658226 24214 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:27.658326 24214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:27.658571 24214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:27.660800 24214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "9d279b28dba34ecd9497de99f7784149"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "fba737c2a21ce7e18145bf142b0ad935"
server_key_iv: "97d663a7a0b0071562a8b6f3e42a00f4"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.661403 24214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "9d279b28dba34ecd9497de99f7784149"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "fba737c2a21ce7e18145bf142b0ad935"
server_key_iv: "97d663a7a0b0071562a8b6f3e42a00f4"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.665331 24214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:12:27.667954 24228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:27.669394 24214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:12:27.669565 24214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "9d279b28dba34ecd9497de99f7784149"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "fba737c2a21ce7e18145bf142b0ad935"
server_key_iv: "97d663a7a0b0071562a8b6f3e42a00f4"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.669713 24214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:27.693859 24214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:27.694644 24214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:27.694887 24214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:27.702514 24214 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:45043
I20260806 16:12:27.702523 24280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:45043 every 8 connection(s)
I20260806 16:12:27.703558 24214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:27.705267 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24214
I20260806 16:12:27.705406 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:27.705681 18792 external_mini_cluster.cc:1428] Setting key d18d1de88836cdcbab6f953e0120f31f
I20260806 16:12:27.707335 24281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:27.714579 24281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149: Bootstrap starting.
I20260806 16:12:27.715503 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:27.707059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.18.90.62:45043)
0806 16:12:27.707710 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:27.707728 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:27.707767 (+ 39us) server_negotiation.cc:407] Connection header received
0806 16:12:27.708508 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:27.708526 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:27.708948 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:27.709306 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:27.709731 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.710735 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.712013 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.712569 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.712710 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.713620 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:27.713625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:27.713650 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:27.714049 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:27.714109 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:27.714129 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:27.714214 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:27.714394 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 16:12:27.714742 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260806 16:12:27.717016 24281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:27.717875 24281 log.cc:824] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:27.718672 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:12:27.720088 24281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149: No bootstrap required, opened a new log
I20260806 16:12:27.723731 24281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d279b28dba34ecd9497de99f7784149" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45043 } }
I20260806 16:12:27.724002 24281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:27.724083 24281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d279b28dba34ecd9497de99f7784149, State: Initialized, Role: FOLLOWER
I20260806 16:12:27.724574 24281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d279b28dba34ecd9497de99f7784149" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45043 } }
I20260806 16:12:27.724753 24281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:27.724826 24281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:27.724933 24281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:27.726056 24281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d279b28dba34ecd9497de99f7784149" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45043 } }
I20260806 16:12:27.726426 24281 leader_election.cc:302] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d279b28dba34ecd9497de99f7784149; no voters:
I20260806 16:12:27.726749 24281 leader_election.cc:288] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:27.726884 24286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:27.727119 24286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [term 1 LEADER]: Becoming Leader. State: Replica: 9d279b28dba34ecd9497de99f7784149, State: Running, Role: LEADER
I20260806 16:12:27.727478 24286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d279b28dba34ecd9497de99f7784149" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45043 } }
I20260806 16:12:27.728060 24281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:27.729215 24288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d279b28dba34ecd9497de99f7784149. Latest consensus state: current_term: 1 leader_uuid: "9d279b28dba34ecd9497de99f7784149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d279b28dba34ecd9497de99f7784149" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45043 } } }
I20260806 16:12:27.729362 24288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:27.729727 24287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d279b28dba34ecd9497de99f7784149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d279b28dba34ecd9497de99f7784149" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45043 } } }
I20260806 16:12:27.729868 24295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:27.729894 24287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:27.733177 24295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:27.738808 24295 catalog_manager.cc:1347] Generated new cluster ID: 764ef3a020ba423cbf45ecc40a7a200a
I20260806 16:12:27.738888 24295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:27.749692 24295 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:27.750655 24295 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:27.756222 24295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d279b28dba34ecd9497de99f7784149: Generated new TSK 0
I20260806 16:12:27.756965 24295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:12:27.773542 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:27.768713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.18.90.62:45043)
0806 16:12:27.768872 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:27.768878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:27.769088 (+ 210us) server_negotiation.cc:407] Connection header received
0806 16:12:27.769299 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:27.769302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:27.769349 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:27.769464 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:12:27.769708 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.770268 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.771165 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.771395 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.771507 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.771837 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:12:27.773145 (+ 1308us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 16:12:27.773388 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 16:12:27.773622 24311 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:27.768640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45043 (local address 127.0.0.1:36744)
0806 16:12:27.768998 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:27.769011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:27.769140 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:27.769463 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:27.769467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:27.769475 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:12:27.769588 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:27.769595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.770391 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.770395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:27.771041 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:27.771050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:27.771521 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:27.771525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:27.771614 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:27.771646 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:12:27.773309 (+ 1663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 16:12:27.773452 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.90.62:45043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
W20260806 16:12:27.774130 24284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260806 16:12:27.776458 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24214
E20260806 16:12:27.785555 24312 webserver.cc:533] Webserver: sslize failed:
E20260806 16:12:27.785641 24312 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T16:12:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T16:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:27Z Disabled control of system clock
I20260806 16:12:27.826733 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45805
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:45805
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/testchainca.cert with env {}
W20260806 16:12:27.940347 24316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:27.940616 24316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:27.940691 24316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:27.944782 24316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:27.944844 24316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:27.944860 24316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:27.944880 24316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:27.944895 24316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:12:27.944953 24316 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:12:27.944978 24316 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 16:12:27.950471 24316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:45805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:27.951646 24316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:27.952807 24316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:27.961544 24321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:27.961618 24316 server_base.cc:1034] running on GCE node
W20260806 16:12:27.961602 24322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:27.961544 24324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:27.962319 24316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:27.963440 24316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:27.964628 24316 hybrid_clock.cc:648] HybridClock initialized: now 1786032747964602 us; error 47 us; skew 500 ppm
I20260806 16:12:27.966471 24316 webserver.cc:460] Webserver started at http://127.18.90.62:43521/ using document root <none> and password file <none>
I20260806 16:12:27.967058 24316 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:27.967154 24316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:27.967391 24316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:27.969507 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "7083a530145945ccb6ef79cafc9bd579"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "dab7ccf50c55b23b18c85a54dbaa327f"
server_key_iv: "75ad2be4edf2aa73beb8bd82537cea9e"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.970088 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "7083a530145945ccb6ef79cafc9bd579"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "dab7ccf50c55b23b18c85a54dbaa327f"
server_key_iv: "75ad2be4edf2aa73beb8bd82537cea9e"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.973788 24316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:12:27.976151 24330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:27.977761 24316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:27.977906 24316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "7083a530145945ccb6ef79cafc9bd579"
format_stamp: "Formatted at 2026-08-06 16:12:27 on dist-test-slave-0b58"
server_key: "dab7ccf50c55b23b18c85a54dbaa327f"
server_key_iv: "75ad2be4edf2aa73beb8bd82537cea9e"
server_key_version: "encryptionkey@0"
I20260806 16:12:27.978055 24316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:27.992686 24316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:27.993371 24316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:27.993566 24316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:28.000854 24316 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:45805
I20260806 16:12:28.000864 24382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:45805 every 8 connection(s)
I20260806 16:12:28.001837 24316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:28.002913 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24316
I20260806 16:12:28.003057 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:28.003348 18792 external_mini_cluster.cc:1428] Setting key f09de6df267f981132e2707ef1801855
I20260806 16:12:28.005124 24383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:28.010946 24383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579: Bootstrap starting.
I20260806 16:12:28.012799 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:28.004702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.18.90.62:45805)
0806 16:12:28.005412 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:28.005429 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:28.005466 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:12:28.006101 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:28.006118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:28.006534 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:28.006874 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:28.007315 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:28.008362 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:28.009303 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:28.009806 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:28.009988 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:28.010933 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:28.010938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:28.010951 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:28.011402 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:28.011461 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:28.011482 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:28.011592 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:28.011719 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 16:12:28.011993 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260806 16:12:28.013568 24383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:28.014361 24383 log.cc:824] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:28.015832 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:12:28.016778 24383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579: No bootstrap required, opened a new log
I20260806 16:12:28.020607 24383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7083a530145945ccb6ef79cafc9bd579" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45805 } }
I20260806 16:12:28.020893 24383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:28.020972 24383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7083a530145945ccb6ef79cafc9bd579, State: Initialized, Role: FOLLOWER
I20260806 16:12:28.021433 24383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7083a530145945ccb6ef79cafc9bd579" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45805 } }
I20260806 16:12:28.021588 24383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:28.021659 24383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:28.021770 24383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:28.022740 24383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7083a530145945ccb6ef79cafc9bd579" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45805 } }
I20260806 16:12:28.023095 24383 leader_election.cc:302] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7083a530145945ccb6ef79cafc9bd579; no voters:
I20260806 16:12:28.023419 24383 leader_election.cc:288] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:28.023554 24388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:28.023854 24388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [term 1 LEADER]: Becoming Leader. State: Replica: 7083a530145945ccb6ef79cafc9bd579, State: Running, Role: LEADER
I20260806 16:12:28.024179 24388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7083a530145945ccb6ef79cafc9bd579" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45805 } }
I20260806 16:12:28.024825 24383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:28.025960 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7083a530145945ccb6ef79cafc9bd579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7083a530145945ccb6ef79cafc9bd579" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45805 } } }
I20260806 16:12:28.026103 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:28.026551 24397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:28.027027 24390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7083a530145945ccb6ef79cafc9bd579. Latest consensus state: current_term: 1 leader_uuid: "7083a530145945ccb6ef79cafc9bd579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7083a530145945ccb6ef79cafc9bd579" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45805 } } }
I20260806 16:12:28.027143 24390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:28.029850 24397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:28.035641 24397 catalog_manager.cc:1347] Generated new cluster ID: 871763c6c40f484fa4a3cdeef2603b03
I20260806 16:12:28.035707 24397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:28.041316 24397 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:28.042493 24397 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:28.050480 24397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7083a530145945ccb6ef79cafc9bd579: Generated new TSK 0
I20260806 16:12:28.051230 24397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:12:28.109695 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:28.064151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.18.90.62:45805)
0806 16:12:28.064327 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:28.064331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:28.064446 (+ 115us) server_negotiation.cc:407] Connection header received
0806 16:12:28.064614 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:28.064616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:28.064664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:28.064753 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:12:28.065031 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:28.065547 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:28.066830 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:28.067033 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:28.067105 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:28.067457 (+ 352us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:12:28.109274 (+ 41817us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 16:12:28.109568 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260806 16:12:28.109833 24386 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53276: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 16:12:28.110124 24413 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:28.064002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45805 (local address 127.0.0.1:53276)
0806 16:12:28.064346 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:28.064361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:28.064483 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:28.064787 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:28.064790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:28.064798 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:12:28.064923 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:28.064930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:28.065659 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:28.065663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:28.066696 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:28.066708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:28.067157 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:28.067161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:28.067260 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:28.067291 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:12:28.109542 (+ 42251us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 16:12:28.109785 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.90.62:45805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260806 16:12:28.110941 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24316
2026-08-06T16:12:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (313 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:28 dist-test-slave-0b58 krb5kdc[24426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:28 dist-test-slave-0b58 krb5kdc[24426](info): set up 2 sockets
Aug 06 16:12:28 dist-test-slave-0b58 krb5kdc[24426](info): commencing operation
krb5kdc: starting...
W20260806 16:12:29.421406 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:29 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:29.552184 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:41797
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:41797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:29.665274 24439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:29.665630 24439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:29.665719 24439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:29.669854 24439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:29.669960 24439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:29.669996 24439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:29.670037 24439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:29.670068 24439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:29.675560 24439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:41797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:41797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:29.676789 24439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:29.677891 24439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:29.686758 24445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:29.686940 24444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:29.686758 24447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:29.687125 24439 server_base.cc:1034] running on GCE node
I20260806 16:12:29.687773 24439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:29.689078 24439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:29.690289 24439 hybrid_clock.cc:648] HybridClock initialized: now 1786032749690273 us; error 31 us; skew 500 ppm
Aug 06 16:12:29 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:29.698459 24439 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:29.699477 24439 webserver.cc:460] Webserver started at http://127.18.90.62:35751/ using document root <none> and password file <none>
I20260806 16:12:29.700089 24439 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:29.700188 24439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:29.700470 24439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:29.702664 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "628bded71fee428996f805e4074887e4"
format_stamp: "Formatted at 2026-08-06 16:12:29 on dist-test-slave-0b58"
server_key: "ef6c81166fb15cd427944889af61cfcf"
server_key_iv: "19d4610ba9f7ec6f71f61a473a7a3fc1"
server_key_version: "encryptionkey@0"
I20260806 16:12:29.703291 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "628bded71fee428996f805e4074887e4"
format_stamp: "Formatted at 2026-08-06 16:12:29 on dist-test-slave-0b58"
server_key: "ef6c81166fb15cd427944889af61cfcf"
server_key_iv: "19d4610ba9f7ec6f71f61a473a7a3fc1"
server_key_version: "encryptionkey@0"
I20260806 16:12:29.707270 24439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:12:29.709996 24454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:29.711326 24439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:12:29.711490 24439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "628bded71fee428996f805e4074887e4"
format_stamp: "Formatted at 2026-08-06 16:12:29 on dist-test-slave-0b58"
server_key: "ef6c81166fb15cd427944889af61cfcf"
server_key_iv: "19d4610ba9f7ec6f71f61a473a7a3fc1"
server_key_version: "encryptionkey@0"
I20260806 16:12:29.711617 24439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:29.754015 24439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:29.757741 24439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:29.757973 24439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:29.765336 24439 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:41797
I20260806 16:12:29.765362 24506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:41797 every 8 connection(s)
I20260806 16:12:29.766357 24439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:29.769210 24507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:29.770053 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24439
I20260806 16:12:29.770198 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:29.770507 18792 external_mini_cluster.cc:1428] Setting key c546ab3c459b76fe0dbe62a3854be5e5
I20260806 16:12:29.775570 24507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4: Bootstrap starting.
I20260806 16:12:29.777930 24507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:29.778949 24507 log.cc:824] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:12:29 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:29.781167 24507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4: No bootstrap required, opened a new log
I20260806 16:12:29.784754 24507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628bded71fee428996f805e4074887e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41797 } }
I20260806 16:12:29.784993 24507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:29.785053 24507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628bded71fee428996f805e4074887e4, State: Initialized, Role: FOLLOWER
I20260806 16:12:29.785470 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:29.772026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.18.90.62:41797)
0806 16:12:29.772529 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:29.772546 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:29.772583 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:12:29.773236 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:29.773264 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:29.773608 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:29.773960 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:29.774809 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:29.775895 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:29.776825 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:29.777314 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:29.777487 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:29.780576 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:29.780582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:29.780596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:29.780622 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:29.782406 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:29.782890 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:29.782895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:29.782900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:29.783011 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:29.783315 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:29.783319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:29.783320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:29.783720 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:29.783944 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:29.784200 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 16:12:29.784530 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":155,"threads_started":1}
I20260806 16:12:29.785504 24507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628bded71fee428996f805e4074887e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41797 } }
I20260806 16:12:29.785666 24507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:29.785770 24507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:29.785895 24507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:29.787025 24507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628bded71fee428996f805e4074887e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41797 } }
I20260806 16:12:29.787387 24507 leader_election.cc:302] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628bded71fee428996f805e4074887e4; no voters:
I20260806 16:12:29.787745 24507 leader_election.cc:288] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:29.787856 24513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:29.788064 24513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [term 1 LEADER]: Becoming Leader. State: Replica: 628bded71fee428996f805e4074887e4, State: Running, Role: LEADER
I20260806 16:12:29.788460 24513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628bded71fee428996f805e4074887e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41797 } }
I20260806 16:12:29.789064 24507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:29.790105 24515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628bded71fee428996f805e4074887e4. Latest consensus state: current_term: 1 leader_uuid: "628bded71fee428996f805e4074887e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628bded71fee428996f805e4074887e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41797 } } }
I20260806 16:12:29.790241 24515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:29.790577 24514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "628bded71fee428996f805e4074887e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628bded71fee428996f805e4074887e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41797 } } }
I20260806 16:12:29.790676 24522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:29.790735 24514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:29.793530 24522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:29.799337 24522 catalog_manager.cc:1347] Generated new cluster ID: 7277a742464649e6bcc1e853df9b7160
I20260806 16:12:29.799420 24522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:29.820647 24522 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:29.821739 24522 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:29.827805 24522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 628bded71fee428996f805e4074887e4: Generated new TSK 0
I20260806 16:12:29.828603 24522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:29.858671 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41797
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:29.970971 24535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:29.971244 24535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:29.971289 24535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:29.975303 24535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:29.975361 24535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:29.975419 24535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:29.980530 24535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:29.981540 24535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:29.982512 24535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:29.993073 24543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:29.993059 24541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:29.993054 24540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:29.993402 24535 server_base.cc:1034] running on GCE node
I20260806 16:12:29.993794 24535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:29.994419 24535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:29.995604 24535 hybrid_clock.cc:648] HybridClock initialized: now 1786032749995577 us; error 49 us; skew 500 ppm
Aug 06 16:12:29 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:29.998384 24535 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:29.999221 24535 webserver.cc:460] Webserver started at http://127.18.90.1:33561/ using document root <none> and password file <none>
I20260806 16:12:29.999791 24535 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:29.999886 24535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:30.000126 24535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:30.002216 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "f5a596e54b8144e5b2265a06a8a8f22c"
format_stamp: "Formatted at 2026-08-06 16:12:29 on dist-test-slave-0b58"
server_key: "fd1c76a158a025fe906344e0c2b3cf3a"
server_key_iv: "4f571819cb840a74b2e745f64ae33629"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.002810 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "f5a596e54b8144e5b2265a06a8a8f22c"
format_stamp: "Formatted at 2026-08-06 16:12:29 on dist-test-slave-0b58"
server_key: "fd1c76a158a025fe906344e0c2b3cf3a"
server_key_iv: "4f571819cb840a74b2e745f64ae33629"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.006599 24535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:30.009094 24550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.010497 24535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:30.010636 24535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "f5a596e54b8144e5b2265a06a8a8f22c"
format_stamp: "Formatted at 2026-08-06 16:12:29 on dist-test-slave-0b58"
server_key: "fd1c76a158a025fe906344e0c2b3cf3a"
server_key_iv: "4f571819cb840a74b2e745f64ae33629"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.010759 24535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:30.030315 24535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:30.033699 24535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:30.033905 24535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:30.034509 24535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:30.035358 24535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:30.035423 24535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.035492 24535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:30.035534 24535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.044498 24535 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:44129
I20260806 16:12:30.044521 24664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:44129 every 8 connection(s)
I20260806 16:12:30.045440 24535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:30.055742 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24535
I20260806 16:12:30.055897 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:30.056233 18792 external_mini_cluster.cc:1428] Setting key d7365c8b728a0fd4ba496ecae899e510
I20260806 16:12:30.059036 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.047262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:43801 (local address 127.18.90.62:41797)
0806 16:12:30.047421 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.047426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.048291 (+ 865us) server_negotiation.cc:407] Connection header received
0806 16:12:30.049663 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.049666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.049713 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.049786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.050936 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.051468 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.052570 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.052796 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.052892 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.055282 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:30.055285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:30.055287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:30.055311 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:30.056994 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.057504 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.057508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.057510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.057593 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.057936 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.057940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.057942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.058121 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:30.058229 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:30.058708 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 16:12:30.058851 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:12:30.059836 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.047678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:41797 (local address 127.18.90.1:43801)
0806 16:12:30.048120 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:30.048169 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:30.049463 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:30.049920 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:30.049926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:30.050282 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:30.050778 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:30.050792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.051576 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.051579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:30.052464 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:30.052473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.052950 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.052953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:30.053101 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.053644 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:30.053660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:30.055121 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:30.057118 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:30.057123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:30.057138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:30.057395 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:30.057693 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:30.057696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:30.057698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:30.057837 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:30.058228 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:30.058235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:30.058478 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 16:12:30.058673 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 16:12:30.058997 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260806 16:12:30.061084 24665 heartbeater.cc:347] Connected to a master server at 127.18.90.62:41797
I20260806 16:12:30.061349 24665 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:30.061930 24665 heartbeater.cc:511] Master 127.18.90.62:41797 requested a full tablet report, sending...
I20260806 16:12:30.063720 24471 ts_manager.cc:194] Registered new tserver with Master: f5a596e54b8144e5b2265a06a8a8f22c (127.18.90.1:44129)
I20260806 16:12:30.065442 24471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:43801
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:30.085121 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41797
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:30.196561 24671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:30.196810 24671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:30.196851 24671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:30.200806 24671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:30.200863 24671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:30.200918 24671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:30.206574 24671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:30.207589 24671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:30.208644 24671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:30.218605 24676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:30.218595 24679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:30.218614 24677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:30.218912 24671 server_base.cc:1034] running on GCE node
I20260806 16:12:30.219280 24671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:30.219894 24671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:30.221045 24671 hybrid_clock.cc:648] HybridClock initialized: now 1786032750221026 us; error 28 us; skew 500 ppm
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.224614 24671 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:30.225790 24671 webserver.cc:460] Webserver started at http://127.18.90.2:43607/ using document root <none> and password file <none>
I20260806 16:12:30.226558 24671 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:30.226629 24671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:30.226903 24671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:30.229696 24671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "30f0788ef2df48f288e6286d4b08f742"
format_stamp: "Formatted at 2026-08-06 16:12:30 on dist-test-slave-0b58"
server_key: "afecbca5399f51d5f5ddd81fcd7d09ea"
server_key_iv: "655cad61a9647f0d75706cc2c940e01c"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.230298 24671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "30f0788ef2df48f288e6286d4b08f742"
format_stamp: "Formatted at 2026-08-06 16:12:30 on dist-test-slave-0b58"
server_key: "afecbca5399f51d5f5ddd81fcd7d09ea"
server_key_iv: "655cad61a9647f0d75706cc2c940e01c"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.234073 24671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:12:30.236526 24686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.237728 24671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:30.237861 24671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "30f0788ef2df48f288e6286d4b08f742"
format_stamp: "Formatted at 2026-08-06 16:12:30 on dist-test-slave-0b58"
server_key: "afecbca5399f51d5f5ddd81fcd7d09ea"
server_key_iv: "655cad61a9647f0d75706cc2c940e01c"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.237970 24671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:30.264016 24671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:30.267156 24671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:30.267385 24671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:30.268013 24671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:30.268944 24671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:30.269014 24671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.269086 24671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:30.269137 24671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.278576 24671 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:34345
I20260806 16:12:30.278600 24802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:34345 every 8 connection(s)
I20260806 16:12:30.279466 24671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:30.282049 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24671
I20260806 16:12:30.282174 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:30.282465 18792 external_mini_cluster.cc:1428] Setting key 85c6968f13b57bffdff7f235e75723c0
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:12:30.294657 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.281308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:55977 (local address 127.18.90.62:41797)
0806 16:12:30.281447 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.281451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.282413 (+ 962us) server_negotiation.cc:407] Connection header received
0806 16:12:30.284217 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.284220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.284289 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.284377 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.285560 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.286292 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.287657 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.287889 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.287997 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.290811 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:30.290815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:30.290818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:30.290844 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:30.292516 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.293024 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.293028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.293030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.293112 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.293451 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.293455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.293457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.293637 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:30.293748 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:30.294354 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 16:12:30.294482 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:12:30.295607 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.281695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:41797 (local address 127.18.90.2:55977)
0806 16:12:30.282260 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:30.282320 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:30.283999 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:30.284557 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:30.284564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:30.284916 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:30.285420 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:30.285434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.286451 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.286457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:30.287521 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:30.287531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.288051 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.288055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:30.288234 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.288931 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:30.288947 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:30.290650 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:30.292632 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:30.292638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:30.292652 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:30.292916 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:30.293202 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:30.293206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:30.293208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:30.293353 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:30.293747 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:30.293765 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:30.294081 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 16:12:30.294307 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 16:12:30.294624 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:12:30.296882 24803 heartbeater.cc:347] Connected to a master server at 127.18.90.62:41797
I20260806 16:12:30.297145 24803 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:30.297762 24803 heartbeater.cc:511] Master 127.18.90.62:41797 requested a full tablet report, sending...
I20260806 16:12:30.298933 24471 ts_manager.cc:194] Registered new tserver with Master: 30f0788ef2df48f288e6286d4b08f742 (127.18.90.2:34345)
I20260806 16:12:30.299793 24471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:55977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:30.313432 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41797
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:30.428086 24808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:30.428442 24808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:30.428493 24808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:30.432930 24808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:30.432988 24808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:30.433079 24808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:30.438283 24808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:30.439359 24808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:30.440441 24808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:30.450086 24813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:30.450052 24814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:30.450059 24816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:30.450346 24808 server_base.cc:1034] running on GCE node
I20260806 16:12:30.450780 24808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:30.451422 24808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:30.452579 24808 hybrid_clock.cc:648] HybridClock initialized: now 1786032750452566 us; error 24 us; skew 500 ppm
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.455412 24808 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:30.456374 24808 webserver.cc:460] Webserver started at http://127.18.90.3:46211/ using document root <none> and password file <none>
I20260806 16:12:30.456943 24808 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:30.457007 24808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:30.457247 24808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:30.459303 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "c215c97f920e431da613e927284ad528"
format_stamp: "Formatted at 2026-08-06 16:12:30 on dist-test-slave-0b58"
server_key: "0dcbbbc87b5640c3673fb3cf13b40d6f"
server_key_iv: "90b321cafb0084389c584c1d0f3a29e8"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.459960 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "c215c97f920e431da613e927284ad528"
format_stamp: "Formatted at 2026-08-06 16:12:30 on dist-test-slave-0b58"
server_key: "0dcbbbc87b5640c3673fb3cf13b40d6f"
server_key_iv: "90b321cafb0084389c584c1d0f3a29e8"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.463646 24808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:12:30.466259 24823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.467434 24808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:12:30.467558 24808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "c215c97f920e431da613e927284ad528"
format_stamp: "Formatted at 2026-08-06 16:12:30 on dist-test-slave-0b58"
server_key: "0dcbbbc87b5640c3673fb3cf13b40d6f"
server_key_iv: "90b321cafb0084389c584c1d0f3a29e8"
server_key_version: "encryptionkey@0"
I20260806 16:12:30.467667 24808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:30.492148 24808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:30.495790 24808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:30.496021 24808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:30.496711 24808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:30.497594 24808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:30.497642 24808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.497711 24808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:30.497743 24808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:30.507165 24808 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:46743
I20260806 16:12:30.507193 24936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:46743 every 8 connection(s)
I20260806 16:12:30.508194 24808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:30.510910 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24808
I20260806 16:12:30.511029 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:30.511370 18792 external_mini_cluster.cc:1428] Setting key 27e191e2517c6ae94d1599e5399e2745
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:30.522920 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.510225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:49059 (local address 127.18.90.62:41797)
0806 16:12:30.510401 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.510406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.511404 (+ 998us) server_negotiation.cc:407] Connection header received
0806 16:12:30.513121 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.513125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.513181 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.513266 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.514612 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.515284 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.516728 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.516965 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.517060 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.519626 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:30.519629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:30.519631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:30.519659 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:30.521041 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.521536 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.521539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.521540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.521617 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.521923 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.521925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.521926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.522100 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:30.522200 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:30.522688 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 16:12:30.522790 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:12:30.523799 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.510640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:41797 (local address 127.18.90.3:49059)
0806 16:12:30.511253 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:30.511300 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:30.512915 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:30.513466 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:30.513475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:30.513856 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:30.514461 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:30.514488 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.515482 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.515486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:30.516601 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:30.516611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.517110 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.517113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:30.517267 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.517915 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:30.517932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:30.519450 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:30.521149 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:30.521154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:30.521166 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:30.521432 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:30.521701 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:30.521703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:30.521704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:30.521844 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:30.522210 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:30.522216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:30.522461 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 16:12:30.522621 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 16:12:30.522892 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":173,"threads_started":1}
I20260806 16:12:30.525166 24937 heartbeater.cc:347] Connected to a master server at 127.18.90.62:41797
I20260806 16:12:30.525441 24937 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:30.525979 24937 heartbeater.cc:511] Master 127.18.90.62:41797 requested a full tablet report, sending...
I20260806 16:12:30.526991 24471 ts_manager.cc:194] Registered new tserver with Master: c215c97f920e431da613e927284ad528 (127.18.90.3:46743)
I20260806 16:12:30.527606 24471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:49059
I20260806 16:12:30.537586 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/test-admin.keytab.
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.554744 18792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 16:12:30.557790 24944 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:30.556265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.18.90.1:44129)
0806 16:12:30.556638 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.556654 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.556678 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:12:30.556732 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.556737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.556936 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.557060 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.557616 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58540: BlockingRecv error: recv got EOF from 127.0.0.1:58540 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":146,"threads_started":1}
W20260806 16:12:30.558320 18792 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc)
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.559369 18792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.560384 18792 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:30.570007 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.561325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.18.90.1:44129)
0806 16:12:30.561467 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.561471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.561566 (+ 95us) server_negotiation.cc:407] Connection header received
0806 16:12:30.561656 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.561659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.561699 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.561766 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.562284 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.563030 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.563843 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.564217 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.564384 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.566709 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:30.566719 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:30.566733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:30.566758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:30.568224 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.568685 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.568690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.568691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.568766 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.568997 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.568999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.569000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.569242 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:30.569406 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:30.569657 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 16:12:30.569831 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 16:12:30.574060 24944 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:30.573129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.18.90.1:44129)
0806 16:12:30.573253 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.573256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.573327 (+ 71us) server_negotiation.cc:407] Connection header received
0806 16:12:30.573439 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.573441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.573475 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.573540 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.573910 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58558: BlockingRecv error: recv got EOF from 127.0.0.1:58558 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 16:12:30.574198 18792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc' of type FILE
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.575248 18792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.576498 18792 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:30.585027 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.577413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.18.90.1:44129)
0806 16:12:30.577543 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.577549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.577652 (+ 103us) server_negotiation.cc:407] Connection header received
0806 16:12:30.577757 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.577760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.577795 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.577862 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.578362 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.578847 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.579598 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.579819 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.579900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.582229 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:30.582232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:30.582234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:30.582254 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:30.583490 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.583873 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.583875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.583877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.583947 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.584172 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.584174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.584175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.584406 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:30.584529 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:30.584714 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 16:12:30.584832 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 16:12:30.587739 24944 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:30.586785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.18.90.1:44129)
0806 16:12:30.586901 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.586907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.587050 (+ 143us) server_negotiation.cc:407] Connection header received
0806 16:12:30.587140 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.587142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.587197 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.587255 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.587594 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60040: BlockingRecv error: recv got EOF from 127.0.0.1:60040 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 16:12:30.587947 18792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032720994684-18792-0/krb5kdc/krb5cc' of type FILE
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.589052 18792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:30.590080 18792 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:12:30.598357 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:30.590929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.18.90.1:44129)
0806 16:12:30.591085 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:30.591088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:30.591155 (+ 67us) server_negotiation.cc:407] Connection header received
0806 16:12:30.591270 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:30.591272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:30.591307 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:30.591373 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:30.591814 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.592362 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.593094 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:30.593292 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:30.593366 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:30.595613 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:30.595616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:30.595618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:30.595639 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:30.596921 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.597299 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.597302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.597303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.597378 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:30.597600 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:30.597602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:30.597603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:30.597754 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:30.597878 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:30.598049 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 16:12:30.598166 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 16:12:30.599474 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24535
I20260806 16:12:30.606324 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24671
I20260806 16:12:30.613032 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24808
I20260806 16:12:30.619253 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24439
2026-08-06T16:12:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2255 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24957](info): set up 2 sockets
Aug 06 16:12:30 dist-test-slave-0b58 krb5kdc[24957](info): commencing operation
krb5kdc: starting...
W20260806 16:12:31.665055 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:31 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab.
Entry for principal oryx/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab.
I20260806 16:12:31.793610 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:32877
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab
--principal=oryx/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:32877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:31.906656 24970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:31.906978 24970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:31.907080 24970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:31.911314 24970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:31.911401 24970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:31.911437 24970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:31.911496 24970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:31.911552 24970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:31.917013 24970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:32877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab
--principal=oryx/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:32877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:31.918272 24970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:31.919346 24970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:31.928301 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:31.928314 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:31.928354 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:31.928472 24970 server_base.cc:1034] running on GCE node
I20260806 16:12:31.929253 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:31.930493 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:31.931692 24970 hybrid_clock.cc:648] HybridClock initialized: now 1786032751931678 us; error 27 us; skew 500 ppm
Aug 06 16:12:31 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:31.934625 24970 init.cc:377] Logged in from keytab as oryx/127.18.90.62@KRBTEST.COM (short username oryx)
I20260806 16:12:31.935624 24970 webserver.cc:460] Webserver started at http://127.18.90.62:41221/ using document root <none> and password file <none>
I20260806 16:12:31.936229 24970 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:31.936374 24970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:31.936619 24970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:31.938778 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "249afc2e239449e1aad233ef197bd6c2"
format_stamp: "Formatted at 2026-08-06 16:12:31 on dist-test-slave-0b58"
server_key: "5c2e2a4b97b7b3a2eb7f2b5ef9653304"
server_key_iv: "af07c3f1e3f94ec2b36976b9ca016ba7"
server_key_version: "encryptionkey@0"
I20260806 16:12:31.939384 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "249afc2e239449e1aad233ef197bd6c2"
format_stamp: "Formatted at 2026-08-06 16:12:31 on dist-test-slave-0b58"
server_key: "5c2e2a4b97b7b3a2eb7f2b5ef9653304"
server_key_iv: "af07c3f1e3f94ec2b36976b9ca016ba7"
server_key_version: "encryptionkey@0"
I20260806 16:12:31.943439 24970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:31.945942 24985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:31.947268 24970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:31.947432 24970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "249afc2e239449e1aad233ef197bd6c2"
format_stamp: "Formatted at 2026-08-06 16:12:31 on dist-test-slave-0b58"
server_key: "5c2e2a4b97b7b3a2eb7f2b5ef9653304"
server_key_iv: "af07c3f1e3f94ec2b36976b9ca016ba7"
server_key_version: "encryptionkey@0"
I20260806 16:12:31.947552 24970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:31.963165 24970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:31.966849 24970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:31.967078 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:31.974792 24970 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:32877
I20260806 16:12:31.975117 25039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:32877 every 8 connection(s)
I20260806 16:12:31.975919 24970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:31.978817 25040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:31.980796 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 24970
I20260806 16:12:31.980966 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:31.981261 18792 external_mini_cluster.cc:1428] Setting key 76040061bd9d9988c1550174d34f192e
I20260806 16:12:31.985922 25040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Bootstrap starting.
I20260806 16:12:31.988816 25040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:31.989849 25040 log.cc:824] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:12:31 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:31.992153 25040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: No bootstrap required, opened a new log
I20260806 16:12:31.997179 25040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } }
I20260806 16:12:31.997463 25040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:31.997512 25040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249afc2e239449e1aad233ef197bd6c2, State: Initialized, Role: FOLLOWER
I20260806 16:12:31.997754 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:31.982677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.18.90.62:32877)
0806 16:12:31.983097 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:31.983130 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:31.983167 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:12:31.983814 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:31.983832 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:31.984187 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:31.984558 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:31.985569 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:31.986651 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:31.987857 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:31.988420 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:31.988544 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:31.991619 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:31.991624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:31.991639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:31.991667 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:31.994150 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:31.994829 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:31.994835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:31.994842 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:31.994997 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:31.995384 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:31.995388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:31.995389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:31.995932 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:31.996127 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:31.996414 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 16:12:31.996737 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":146,"threads_started":1}
I20260806 16:12:31.998013 25040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } }
I20260806 16:12:31.998178 25040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:31.998270 25040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:31.998374 25040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:31.999522 25040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } }
I20260806 16:12:31.999929 25040 leader_election.cc:302] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249afc2e239449e1aad233ef197bd6c2; no voters:
I20260806 16:12:32.000365 25040 leader_election.cc:288] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:32.000491 25045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:32.000734 25045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 LEADER]: Becoming Leader. State: Replica: 249afc2e239449e1aad233ef197bd6c2, State: Running, Role: LEADER
I20260806 16:12:32.001096 25045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } }
I20260806 16:12:32.001684 25040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:32.002774 25047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249afc2e239449e1aad233ef197bd6c2. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } }
I20260806 16:12:32.003017 25046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } }
I20260806 16:12:32.003125 25047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:32.003232 25046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
W20260806 16:12:32.006664 25061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:12:32.006744 25061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:12:32.006826 25055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:32.007540 25055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:32.013734 25055 catalog_manager.cc:1347] Generated new cluster ID: 39ab755326094acd89fc86cbad146351
I20260806 16:12:32.013815 25055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:32.030268 25055 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:32.031250 25055 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:32.035198 25055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Generated new TSK 0
I20260806 16:12:32.035902 25055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab.
Entry for principal oryx/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab.
I20260806 16:12:32.067301 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:32877
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:32.186652 25066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:32.186947 25066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:32.186993 25066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:32.191035 25066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:32.191104 25066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:32.191162 25066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:32.196631 25066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:32877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:32.197779 25066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:32.198828 25066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:32.208756 25074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.208902 25066 server_base.cc:1034] running on GCE node
W20260806 16:12:32.208766 25072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:32.208756 25071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.209535 25066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:32.210175 25066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:32.211377 25066 hybrid_clock.cc:648] HybridClock initialized: now 1786032752211367 us; error 29 us; skew 500 ppm
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:32.214145 25066 init.cc:377] Logged in from keytab as oryx/127.18.90.1@KRBTEST.COM (short username oryx)
I20260806 16:12:32.215066 25066 webserver.cc:460] Webserver started at http://127.18.90.1:36337/ using document root <none> and password file <none>
I20260806 16:12:32.215649 25066 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:32.215749 25066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:32.215991 25066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:32.218145 25066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "870db067f3704aceba4f9bae8a294d1c"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a0940c67f0e71fdef0853952e8edd8d8"
server_key_iv: "4bc18ce92921f83f800e3245bf8403be"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.218839 25066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "870db067f3704aceba4f9bae8a294d1c"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a0940c67f0e71fdef0853952e8edd8d8"
server_key_iv: "4bc18ce92921f83f800e3245bf8403be"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.222995 25066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:12:32.226550 25081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.228068 25066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:32.228237 25066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "870db067f3704aceba4f9bae8a294d1c"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a0940c67f0e71fdef0853952e8edd8d8"
server_key_iv: "4bc18ce92921f83f800e3245bf8403be"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.228394 25066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:32.240423 25066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:32.243335 25066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:32.243553 25066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:32.244169 25066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:32.245019 25066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:32.245090 25066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.245160 25066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:32.245225 25066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.254015 25066 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:45227
I20260806 16:12:32.254053 25195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:45227 every 8 connection(s)
I20260806 16:12:32.254957 25066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:32.264838 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25066
I20260806 16:12:32.264967 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:32.265277 18792 external_mini_cluster.cc:1428] Setting key 8abe264ddacd35f4daaf1378c2c7f2f2
I20260806 16:12:32.269627 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.256885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:58789 (local address 127.18.90.62:32877)
0806 16:12:32.257055 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.257060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.257928 (+ 868us) server_negotiation.cc:407] Connection header received
0806 16:12:32.259263 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.259266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.259314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.259388 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.260664 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.261214 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.262400 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.262631 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.262709 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.265381 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.265385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.265388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.265419 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.267062 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.267729 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.267732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.267733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.267815 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.268222 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.268224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.268225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.268456 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.268534 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.269247 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.269399 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:12:32.270419 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.257314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:58789)
0806 16:12:32.257772 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.257818 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.259062 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.259561 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.259567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.259932 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.260512 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.260529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.261362 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.261365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.262283 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.262293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.262771 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.262773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.262925 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.263523 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.263540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.265185 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.267181 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.267189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.267211 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.267619 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.267912 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.267918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.267923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.268125 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.268561 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.268571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.268933 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.269178 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.269473 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
I20260806 16:12:32.271720 25196 heartbeater.cc:347] Connected to a master server at 127.18.90.62:32877
I20260806 16:12:32.271981 25196 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:32.272594 25196 heartbeater.cc:511] Master 127.18.90.62:32877 requested a full tablet report, sending...
I20260806 16:12:32.274149 25003 ts_manager.cc:194] Registered new tserver with Master: 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227)
I20260806 16:12:32.275357 25003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:58789
WARNING: no policy specified for oryx/127.18.90.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab.
Entry for principal oryx/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab.
I20260806 16:12:32.295912 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:32877
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:32.409575 25201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:32.409868 25201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:32.409942 25201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:32.414093 25201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:32.414157 25201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:32.414244 25201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:32.419866 25201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:32877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:32.421070 25201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:32.422173 25201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:32.432015 25207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.432171 25201 server_base.cc:1034] running on GCE node
W20260806 16:12:32.432045 25209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:32.432019 25206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.432876 25201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:32.433540 25201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:32.434727 25201 hybrid_clock.cc:648] HybridClock initialized: now 1786032752434690 us; error 47 us; skew 500 ppm
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:32.437525 25201 init.cc:377] Logged in from keytab as oryx/127.18.90.2@KRBTEST.COM (short username oryx)
I20260806 16:12:32.438452 25201 webserver.cc:460] Webserver started at http://127.18.90.2:36467/ using document root <none> and password file <none>
I20260806 16:12:32.439033 25201 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:32.439126 25201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:32.439370 25201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:32.441499 25201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "a5b3b108823940d2b7685eda5831fbab"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "50c38e6abf22e63b09e060422c899dd2"
server_key_iv: "4f17f52cf4654c46d296815a33a59083"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.442066 25201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "a5b3b108823940d2b7685eda5831fbab"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "50c38e6abf22e63b09e060422c899dd2"
server_key_iv: "4f17f52cf4654c46d296815a33a59083"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.445888 25201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:32.448446 25216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.449681 25201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:32.449822 25201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "a5b3b108823940d2b7685eda5831fbab"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "50c38e6abf22e63b09e060422c899dd2"
server_key_iv: "4f17f52cf4654c46d296815a33a59083"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.449935 25201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:32.467185 25201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:32.470309 25201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:32.470515 25201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:32.471136 25201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:32.471998 25201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:32.472045 25201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.472115 25201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:32.472147 25201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.481003 25201 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38549
I20260806 16:12:32.481020 25330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38549 every 8 connection(s)
I20260806 16:12:32.481956 25201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:32.483245 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25201
I20260806 16:12:32.483378 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:32.483686 18792 external_mini_cluster.cc:1428] Setting key 7ae9a4409508cc1123ca4a6806a3b7f8
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.3@KRBTEST.COM" created.
I20260806 16:12:32.498674 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.483969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:40267 (local address 127.18.90.62:32877)
0806 16:12:32.484147 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.484152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.485265 (+ 1113us) server_negotiation.cc:407] Connection header received
0806 16:12:32.487315 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.487318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.487363 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.487429 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.488833 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.489400 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.491290 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.491512 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.491596 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.494875 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.494878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.494880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.494907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.496492 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.496997 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.497003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.497008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.497097 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.497450 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.497452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.497453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.497637 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.497745 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.498426 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.498545 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:12:32.499956 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.484537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:40267)
0806 16:12:32.485061 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.485140 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.487077 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.487656 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.487670 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.488094 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.488668 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.488682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.489544 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.489550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.491143 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.491159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.491690 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.491696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.491964 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.492690 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.492715 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.494650 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.496600 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.496605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.496619 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.496876 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.497210 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.497214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.497215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.497357 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.497749 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.497766 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.498098 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.498351 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.498749 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260806 16:12:32.501267 25331 heartbeater.cc:347] Connected to a master server at 127.18.90.62:32877
I20260806 16:12:32.501549 25331 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:32.502220 25331 heartbeater.cc:511] Master 127.18.90.62:32877 requested a full tablet report, sending...
I20260806 16:12:32.503409 25003 ts_manager.cc:194] Registered new tserver with Master: a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549)
I20260806 16:12:32.504371 25003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:40267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab.
Entry for principal oryx/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab.
I20260806 16:12:32.518831 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:32877
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:32.634442 25336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:32.634720 25336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:32.634799 25336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:32.638924 25336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:32.638994 25336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:32.639094 25336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:32.644693 25336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:32877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:32.645895 25336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:32.646989 25336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:32.657284 25342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:32.657315 25344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.657377 25336 server_base.cc:1034] running on GCE node
W20260806 16:12:32.657302 25341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.657869 25336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:32.658512 25336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:32.659677 25336 hybrid_clock.cc:648] HybridClock initialized: now 1786032752659663 us; error 25 us; skew 500 ppm
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:32.662472 25336 init.cc:377] Logged in from keytab as oryx/127.18.90.3@KRBTEST.COM (short username oryx)
I20260806 16:12:32.663354 25336 webserver.cc:460] Webserver started at http://127.18.90.3:44855/ using document root <none> and password file <none>
I20260806 16:12:32.663939 25336 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:32.664032 25336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:32.664325 25336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:32.666483 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "748c8722e114457bbb7fdd64d4548e0f"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "79a4e218a72749073ad0bf02201ff717"
server_key_iv: "b54170ede2246c8a002d14a6fe718bed"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.667102 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "748c8722e114457bbb7fdd64d4548e0f"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "79a4e218a72749073ad0bf02201ff717"
server_key_iv: "b54170ede2246c8a002d14a6fe718bed"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.670987 25336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260806 16:12:32.673477 25351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.674784 25336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:32.674925 25336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "748c8722e114457bbb7fdd64d4548e0f"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "79a4e218a72749073ad0bf02201ff717"
server_key_iv: "b54170ede2246c8a002d14a6fe718bed"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.675043 25336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:32.686007 25336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:32.689687 25336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:32.689855 25336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:32.690398 25336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:32.691187 25336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:32.691236 25336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.691275 25336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:32.691291 25336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.700117 25336 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:39809
I20260806 16:12:32.700137 25465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:39809 every 8 connection(s)
I20260806 16:12:32.701052 25336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:32.706287 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25336
I20260806 16:12:32.706430 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:32.706753 18792 external_mini_cluster.cc:1428] Setting key 538ec8328d0d632d10fa95280a35dd3d
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:32.715184 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.702988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:59033 (local address 127.18.90.62:32877)
0806 16:12:32.703132 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.703135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.704001 (+ 866us) server_negotiation.cc:407] Connection header received
0806 16:12:32.705433 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.705436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.705496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.705578 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.707170 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.707956 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.709413 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.709632 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.709716 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.712070 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.712073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.712075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.712101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.713449 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.713912 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.713914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.713915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.713988 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.714271 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.714273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.714274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.714445 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.714520 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.714964 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.715063 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:12:32.716012 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.703388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:59033)
0806 16:12:32.703849 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.703892 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.705200 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.705758 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.705765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.706247 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.706990 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.707011 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.708115 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.708118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.709298 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.709308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.709760 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.709762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.709900 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.710448 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.710464 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.711921 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.713550 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.713554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.713566 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.713826 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.714064 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.714066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.714068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.714193 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.714530 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.714535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.714755 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.714917 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.715159 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":149,"threads_started":1}
I20260806 16:12:32.717216 25466 heartbeater.cc:347] Connected to a master server at 127.18.90.62:32877
I20260806 16:12:32.717468 25466 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:32.717950 25466 heartbeater.cc:511] Master 127.18.90.62:32877 requested a full tablet report, sending...
I20260806 16:12:32.718961 25003 ts_manager.cc:194] Registered new tserver with Master: 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809)
I20260806 16:12:32.719744 25003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:59033
I20260806 16:12:32.722062 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.90.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.61.keytab.
Entry for principal oryx/127.18.90.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.61.keytab.
I20260806 16:12:32.750608 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:46489
--webserver_interface=127.18.90.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.61.keytab
--principal=oryx/127.18.90.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:32877,127.18.90.61:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:32.863514 25471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:32.863787 25471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:32.863835 25471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:32.867762 25471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:32.867820 25471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:32.867839 25471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:32.867856 25471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:32.867871 25471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:32.873122 25471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:32877,127.18.90.61:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.61.keytab
--principal=oryx/127.18.90.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.90.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
W20260806 16:12:32.873333 25471 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.90.62:32877,127.18.90.61:46489'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 16:12:32.874199 25471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:32.875195 25471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:32.883742 25476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.883858 25471 server_base.cc:1034] running on GCE node
W20260806 16:12:32.883760 25477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:32.883742 25479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:32.884667 25471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:32.885813 25471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:32.887041 25471 hybrid_clock.cc:648] HybridClock initialized: now 1786032752887031 us; error 31 us; skew 500 ppm
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:32.889827 25471 init.cc:377] Logged in from keytab as oryx/127.18.90.61@KRBTEST.COM (short username oryx)
I20260806 16:12:32.890789 25471 webserver.cc:460] Webserver started at http://127.18.90.61:45101/ using document root <none> and password file <none>
I20260806 16:12:32.891360 25471 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:32.891454 25471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:32.891698 25471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:32.893849 25471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/instance:
uuid: "a7348509d4614dfbb816db230c4229bb"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a83fc29c87f7cd7594ccec64bd798282"
server_key_iv: "147d9c4e01083ca34015e816cbc1841c"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.894443 25471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal/instance:
uuid: "a7348509d4614dfbb816db230c4229bb"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a83fc29c87f7cd7594ccec64bd798282"
server_key_iv: "147d9c4e01083ca34015e816cbc1841c"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.898312 25471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:32.900885 25486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:32.902189 25471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:32.902331 25471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
uuid: "a7348509d4614dfbb816db230c4229bb"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a83fc29c87f7cd7594ccec64bd798282"
server_key_iv: "147d9c4e01083ca34015e816cbc1841c"
server_key_version: "encryptionkey@0"
I20260806 16:12:32.902454 25471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:32.924875 25471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:32.928206 25471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:32.928452 25471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:32.935577 25471 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.61:46489
I20260806 16:12:32.935883 25538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.61:46489 every 8 connection(s)
I20260806 16:12:32.936705 25471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
I20260806 16:12:32.937832 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25471
I20260806 16:12:32.939999 25539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:32.944607 25539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.61@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:32.954107 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.939342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.18.90.61:46489)
0806 16:12:32.939752 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.939771 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.939930 (+ 159us) server_negotiation.cc:407] Connection header received
0806 16:12:32.940854 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.940878 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.941291 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.941656 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.942384 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.943874 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.944797 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.945462 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.945616 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.948619 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.948625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.948636 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.948666 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.950642 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.951207 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.951213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.951219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.951342 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.951647 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.951649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.951651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.952411 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.952616 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.952864 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.953161 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260806 16:12:32.956426 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.946575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:33848)
0806 16:12:32.946955 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.946993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.947134 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.947443 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.947448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.947821 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.947999 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.948009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.949067 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.949073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.949916 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.949926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.950505 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.950507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.950587 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.951082 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.951099 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.952841 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.954698 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.954705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.954710 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.954987 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.955259 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.955262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.955264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.955354 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.955788 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.955796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.955906 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.956045 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.956221 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260806 16:12:32.956418 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.946629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.18.90.62:32877)
0806 16:12:32.946794 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.946798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.947075 (+ 277us) server_negotiation.cc:407] Connection header received
0806 16:12:32.947244 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.947252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.947307 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.947395 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.948147 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.948926 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.950038 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.950378 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.950496 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.952986 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.952990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.952993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.953022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.954599 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.955081 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.955083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.955085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.955159 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.955447 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.955451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.955453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.955670 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.955780 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.956100 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.956219 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:12:32.957593 25539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:12:32 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032752, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.61@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
I20260806 16:12:32.964998 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.958071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59100)
0806 16:12:32.958177 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.958189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.958248 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.958441 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.958444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.958565 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.958648 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.958652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.959660 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.959662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.960158 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.960164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.960640 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.960642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.960714 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.961165 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.961176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.962286 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.963689 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.963691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.963693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.963882 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.964110 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.964112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.964114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.964183 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.964562 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.964564 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.964637 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.964721 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.964842 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 16:12:32.965016 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.958096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.18.90.61:46489)
0806 16:12:32.958204 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.958208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.958221 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:32.958315 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.958317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.958349 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.958415 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.958740 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.959557 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.960297 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.960546 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.960625 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.962403 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.962406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.962408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.962428 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.963592 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.963963 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.963966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.963967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.964034 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.964310 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.964313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.964315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.964478 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.964557 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.964780 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.964897 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":36}
I20260806 16:12:32.968031 25539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Bootstrap starting.
I20260806 16:12:32.970453 25539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:32.971179 25539 log.cc:824] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:32.973177 25539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: No bootstrap required, opened a new log
I20260806 16:12:32.976143 25539 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } }
I20260806 16:12:32.976441 25539 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:32.976529 25539 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7348509d4614dfbb816db230c4229bb, State: Initialized, Role: FOLLOWER
I20260806 16:12:32.977010 25539 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } }
I20260806 16:12:32.977515 25545 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } } }
I20260806 16:12:32.977663 25545 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:32.978269 25539 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:12:32.983098 25556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:12:32.983187 25556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:12:32.985074 25471 master_runner.cc:428] Detected that this master a7348509d4614dfbb816db230c4229bb is joining an existing cluster
I20260806 16:12:32.985131 25471 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.90.61:46489
I20260806 16:12:32.996695 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.987311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:33860)
0806 16:12:32.988004 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.988017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.988146 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.988544 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.988547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.988876 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.989053 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.989061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.990117 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.990120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.991076 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.991086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.991637 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.991639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.991718 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.992306 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.992326 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.992787 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.994873 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.994877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.994879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.995155 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.995441 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.995446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.995534 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.996011 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.996013 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.996110 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.996347 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.996497 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":27,"spinlock_wait_cycles":81280,"thread_start_us":94,"threads_started":1}
I20260806 16:12:32.996897 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.987815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.18.90.61:46489)
0806 16:12:32.987913 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.987918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.988165 (+ 247us) server_negotiation.cc:407] Connection header received
0806 16:12:32.988356 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.988362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.988407 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.988454 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.989184 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.989960 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.991214 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.991536 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.991606 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.993145 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.993149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.993153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.993180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.994888 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995368 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.995372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.995374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.995450 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995693 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.995699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.995701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.995904 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.995997 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.996538 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.996697 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 16:12:32.997287 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.987751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59108)
0806 16:12:32.988087 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:32.988102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:32.988219 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:32.988573 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:32.988576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:32.988892 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:32.989070 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.989078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.990090 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.990093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.991092 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:32.991103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.991734 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.991738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:32.991840 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.992487 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:32.992503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:32.993006 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:32.995018 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.995025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.995269 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.995530 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:32.995534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:32.995619 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:32.996034 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:32.996039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:32.996133 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 16:12:32.996384 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 16:12:32.997132 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":87,"thread_start_us":92,"threads_started":1}
I20260806 16:12:32.998157 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:32.987384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.18.90.62:32877)
0806 16:12:32.987498 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:32.987503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:32.988080 (+ 577us) server_negotiation.cc:407] Connection header received
0806 16:12:32.988278 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:32.988282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:32.988336 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:32.988393 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:32.989182 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.989988 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.991218 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:32.991441 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:32.991498 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:32.992933 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:32.992937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:32.992940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:32.992966 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:32.994748 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995264 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.995267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.995269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.995361 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:32.995619 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:32.995623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:32.995625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:32.995861 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:32.995962 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:32.997877 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0806 16:12:32.998008 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.62@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
I20260806 16:12:33.012714 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.002616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.18.90.61:46489)
0806 16:12:33.002743 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.002747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.003108 (+ 361us) server_negotiation.cc:407] Connection header received
0806 16:12:33.003351 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.003357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.003406 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.003472 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.004240 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.004771 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.006067 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.006378 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.006490 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.008904 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.008908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.008910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.008935 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.010699 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.011325 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.011331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.011334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.011425 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.011742 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.011747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.011749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.011955 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.012066 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.012435 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.012552 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:12:33.012746 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.002576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59124)
0806 16:12:33.002966 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.003024 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.003157 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.003531 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.003537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.003997 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.004129 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.004137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.004888 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.004893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.005934 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.005945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.006496 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.006499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.006628 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.007218 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.007230 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.008765 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.010846 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.010859 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.010868 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.011203 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.011537 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.011539 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.011540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.011645 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.012067 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.012075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.012205 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.012382 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.012583 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260806 16:12:33.014492 25003 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.90.61:46489: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a7348509d4614dfbb816db230c4229bb"
member_type: NON_VOTER
last_known_addr {
host: "127.18.90.61"
port: 46489
}
attrs {
promote: true
}
}
dest_uuid: "249afc2e239449e1aad233ef197bd6c2"
cas_config_opid_index: -1
I20260806 16:12:33.015142 25003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } }
I20260806 16:12:33.016839 25567 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a7348509d4614dfbb816db230c4229bb (127.18.90.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } }
I20260806 16:12:33.017784 25513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.18.90.60@KRBTEST.COM; defaulting to no policy
I20260806 16:12:33.018872 25567 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.90.61:46489
I20260806 16:12:33.019228 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } }
I20260806 16:12:33.019368 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:33.019454 25471 master.cc:537] Master@127.18.90.61:46489 shutting down...
I20260806 16:12:33.020462 25513 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Refusing update from remote peer 249afc2e239449e1aad233ef197bd6c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.60@KRBTEST.COM" created.
I20260806 16:12:33.021199 25568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 16:12:33.023272 24989 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 16:12:33.024246 24989 consensus_peers.cc:487] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 -> Peer a7348509d4614dfbb816db230c4229bb (127.18.90.61:46489): Couldn't send request to peer a7348509d4614dfbb816db230c4229bb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 16:12:33.024861 25471 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:33.025166 25471 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:33.025291 25471 tablet_replica.cc:330] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: stopping tablet replica
I20260806 16:12:33.029626 25471 master.cc:552] Master@127.18.90.61:46489 shutdown complete.
I20260806 16:12:33.029695 25471 master_runner.cc:305] Clearing existing system tablet
I20260806 16:12:33.029932 25471 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.030004 25471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.031077 25471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 16:12:33.032891 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.033200 25471 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.033351 25471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:33.033396 25471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
uuid: "a7348509d4614dfbb816db230c4229bb"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a83fc29c87f7cd7594ccec64bd798282"
server_key_iv: "147d9c4e01083ca34015e816cbc1841c"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.035472 25471 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:33.037829 25471 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 16:12:33.037909 25471 log.cc:1196] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 16:12:33.038196 25471 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Deleting consensus metadata
I20260806 16:12:33.038386 25471 master_runner.cc:315] Copying system tablet from 127.18.90.62:32877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.60.keytab.
Entry for principal oryx/127.18.90.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.60.keytab.
I20260806 16:12:33.039688 25471 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: tablet copy: Beginning tablet copy session from remote peer at address 127.18.90.62:32877
I20260806 16:12:33.039791 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:35739
--webserver_interface=127.18.90.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.60.keytab
--principal=oryx/127.18.90.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:12:33.049114 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.040721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.18.90.62:32877)
0806 16:12:33.040876 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.040882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.041082 (+ 200us) server_negotiation.cc:407] Connection header received
0806 16:12:33.041236 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.041241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.041301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.041371 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.041911 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.042836 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.044034 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.044377 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.044494 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.045671 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.045675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.045677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.045708 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.047333 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.047781 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.047788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.047789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.047882 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.048162 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.048166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.048168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.048448 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.048558 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.048823 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.048937 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:12:33.049523 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.040633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:33876)
0806 16:12:33.040986 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.040998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.041132 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.041507 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.041510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.041680 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.041773 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.041779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.042972 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.042976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.043896 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.043908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.044512 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.044516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.044606 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.045171 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.045183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.045507 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.047461 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.047464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.047466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.047676 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.047976 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.047979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.047981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.048068 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.048548 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.048551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.048657 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.048773 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.048933 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260806 16:12:33.050155 25023 tablet_copy_service.cc:140] P 249afc2e239449e1aad233ef197bd6c2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a7348509d4614dfbb816db230c4229bb ({username='oryx', principal='oryx/127.18.90.61@KRBTEST.COM'} at 127.0.0.1:33876)
I20260806 16:12:33.050354 25023 tablet_copy_service.cc:161] P 249afc2e239449e1aad233ef197bd6c2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a7348509d4614dfbb816db230c4229bb at {username='oryx', principal='oryx/127.18.90.61@KRBTEST.COM'} at 127.0.0.1:33876: session id = a7348509d4614dfbb816db230c4229bb-00000000000000000000000000000000
I20260806 16:12:33.051990 25023 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 16:12:33.053663 25471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 16:12:33.055696 25471 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: tablet copy: Starting download of 0 data blocks...
I20260806 16:12:33.055850 25471 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: tablet copy: Starting download of 1 WAL segments...
I20260806 16:12:33.057561 25471 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 16:12:33.059137 25023 tablet_copy_service.cc:342] P 249afc2e239449e1aad233ef197bd6c2: Request end of tablet copy session a7348509d4614dfbb816db230c4229bb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.90.61@KRBTEST.COM'} at 127.0.0.1:33876
I20260806 16:12:33.059273 25023 tablet_copy_service.cc:433] P 249afc2e239449e1aad233ef197bd6c2: ending tablet copy session a7348509d4614dfbb816db230c4229bb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a7348509d4614dfbb816db230c4229bb
I20260806 16:12:33.066610 25471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:33.068300 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.068465 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.068681 25471 server_base.cc:1034] running on GCE node
W20260806 16:12:33.068785 25583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.068985 25471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:33.069344 25471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:33.070488 25471 hybrid_clock.cc:648] HybridClock initialized: now 1786032753070502 us; error 47 us; skew 500 ppm
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:33.072026 25471 init.cc:377] Logged in from keytab as oryx/127.18.90.61@KRBTEST.COM (short username oryx)
I20260806 16:12:33.072553 25471 webserver.cc:460] Webserver started at http://127.18.90.61:44821/ using document root <none> and password file <none>
I20260806 16:12:33.072755 25471 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.072813 25471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.073925 25471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:33.074895 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.075140 25471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:33.075238 25471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
uuid: "a7348509d4614dfbb816db230c4229bb"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a83fc29c87f7cd7594ccec64bd798282"
server_key_iv: "147d9c4e01083ca34015e816cbc1841c"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.075352 25471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.088243 25471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:33.090401 25471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:33.095342 25471 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.61:46489
I20260806 16:12:33.095866 25471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
I20260806 16:12:33.096360 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.61:46489 every 8 connection(s)
I20260806 16:12:33.097640 25646 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:12:33.098467 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Bootstrap starting.
I20260806 16:12:33.107290 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:12:33.107611 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: Bootstrap complete.
I20260806 16:12:33.108139 25646 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } }
I20260806 16:12:33.108307 25646 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7348509d4614dfbb816db230c4229bb, State: Initialized, Role: LEARNER
I20260806 16:12:33.108438 25646 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } }
I20260806 16:12:33.108815 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } }
I20260806 16:12:33.108987 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: LEARNER
I20260806 16:12:33.108995 25646 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:33.112010 25660 catalog_manager.cc:1259] Loaded cluster ID: 39ab755326094acd89fc86cbad146351
I20260806 16:12:33.112118 25660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: loading cluster ID for follower catalog manager: success
I20260806 16:12:33.113854 25660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: acquiring CA information for follower catalog manager: success
I20260806 16:12:33.114959 25660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 16:12:33.200093 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:33.200450 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:33.200556 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:33.204914 25581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:33.205001 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:33.205041 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:33.205098 25581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:33.205145 25581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:33.211160 25581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.60.keytab
--principal=oryx/127.18.90.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:35739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.90.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:33.212373 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:33.213451 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:33.221855 25666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.221868 25668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.221853 25665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.222224 25581 server_base.cc:1034] running on GCE node
I20260806 16:12:33.222884 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:33.223977 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:33.225142 25581 hybrid_clock.cc:648] HybridClock initialized: now 1786032753225112 us; error 42 us; skew 500 ppm
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:33.227938 25581 init.cc:377] Logged in from keytab as oryx/127.18.90.60@KRBTEST.COM (short username oryx)
I20260806 16:12:33.228859 25581 webserver.cc:460] Webserver started at http://127.18.90.60:37333/ using document root <none> and password file <none>
I20260806 16:12:33.229478 25581 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.229547 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.229732 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:33.231781 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/instance:
uuid: "e5fe45c2c8f4406a8cf84e1adc035c35"
format_stamp: "Formatted at 2026-08-06 16:12:33 on dist-test-slave-0b58"
server_key: "e53c4bbf0043a0966d68b2c5546ce130"
server_key_iv: "dff3dd82fa62ad35f8c119c54ed6ed42"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.232395 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal/instance:
uuid: "e5fe45c2c8f4406a8cf84e1adc035c35"
format_stamp: "Formatted at 2026-08-06 16:12:33 on dist-test-slave-0b58"
server_key: "e53c4bbf0043a0966d68b2c5546ce130"
server_key_iv: "dff3dd82fa62ad35f8c119c54ed6ed42"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.236358 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:12:33.238873 25675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.240304 25581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:33.240455 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
uuid: "e5fe45c2c8f4406a8cf84e1adc035c35"
format_stamp: "Formatted at 2026-08-06 16:12:33 on dist-test-slave-0b58"
server_key: "e53c4bbf0043a0966d68b2c5546ce130"
server_key_iv: "dff3dd82fa62ad35f8c119c54ed6ed42"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.240576 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.252820 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:33.256182 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:33.256462 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:33.263586 25581 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.60:35739
I20260806 16:12:33.263876 25728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.60:35739 every 8 connection(s)
I20260806 16:12:33.264750 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
I20260806 16:12:33.267601 25729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:33.269595 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25581
I20260806 16:12:33.271710 25729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:12:33.277892 25196 heartbeater.cc:503] Master 127.18.90.62:32877 was elected leader, sending a full tablet report...
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.90.60@KRBTEST.COM
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.60@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:33.284162 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.274374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.18.90.62:32877)
0806 16:12:33.274500 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.274505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.274684 (+ 179us) server_negotiation.cc:407] Connection header received
0806 16:12:33.274815 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.274818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.274879 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.274945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.275805 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.276402 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.277484 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.277742 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.277862 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.281001 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.281005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.281008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.281033 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.282306 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.282779 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.282783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.282785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.282873 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.283198 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.283204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.283209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.283396 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.283504 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.283870 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.283989 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:12:33.285065 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.271138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.18.90.60:35739)
0806 16:12:33.271569 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.271588 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.271630 (+ 42us) server_negotiation.cc:407] Connection header received
0806 16:12:33.272579 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.272605 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.273038 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.273399 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.274189 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.275687 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.276738 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.277216 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.277339 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.280447 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.280452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.280481 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.280511 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.282350 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.282800 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.282806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.282808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.282919 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.283212 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.283215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.283217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.283571 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.283773 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.284030 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.284227 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260806 16:12:33.285064 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.274159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:33880)
0806 16:12:33.274541 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.274588 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.274718 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.275017 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.275023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.275453 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.275681 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.275688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.276538 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.276542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.277340 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.277351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.277903 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.277908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.278040 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.278629 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.278648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.280839 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.282419 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.282429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.282439 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.282672 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.282975 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.282978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.282980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.283101 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.283505 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.283512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.283653 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.283825 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.284151 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":36,"thread_start_us":114,"threads_started":1}
I20260806 16:12:33.286928 25729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.60@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
I20260806 16:12:33.294752 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.287417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59136)
0806 16:12:33.287533 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.287547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.287617 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.288029 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.288031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.288159 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.288335 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.288342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.289193 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.289195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.289734 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.289741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.290153 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.290156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.290250 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.290730 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.290741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.291926 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.293533 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.293536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.293537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.293720 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.293963 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.293965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.293966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.294035 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.294377 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.294379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.294455 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.294533 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.294632 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 16:12:33.294865 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.287543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.18.90.61:46489)
0806 16:12:33.287823 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.287827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.287838 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:12:33.287876 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.287878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.287915 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.287991 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.288424 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.289103 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.289835 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.290060 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.290136 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.292061 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.292065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.292067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.292090 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.293434 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.293804 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.293806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.293807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.293885 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.294109 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.294111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.294112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.294298 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.294374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.294591 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.294696 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":128,"threads_started":1}
I20260806 16:12:33.295648 25729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.60@KRBTEST.COM for oryx/127.18.90.60@KRBTEST.COM
I20260806 16:12:33.303258 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.296153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60950)
0806 16:12:33.296314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.296329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.296427 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.296652 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.296655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.296814 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.296871 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.296876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.297591 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.297597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.298095 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.298102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.298593 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.298597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.298670 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.299139 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.299159 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.300408 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.301828 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.301834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.301837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.302053 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.302326 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.302329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.302331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.302409 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.302781 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.302784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.302862 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.303002 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.303129 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 16:12:33.303383 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.296212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.18.90.60:35739)
0806 16:12:33.296338 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.296345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.296378 (+ 33us) server_negotiation.cc:407] Connection header received
0806 16:12:33.296522 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.296525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.296566 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.296643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.296978 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.297478 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.298210 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.298485 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.298593 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.300529 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.300532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.300535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.300563 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.301722 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.302150 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.302155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.302157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.302231 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.302494 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.302499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.302501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.302670 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.302792 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.303105 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.303210 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 16:12:33.306710 25729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Bootstrap starting.
I20260806 16:12:33.309264 25729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:33.310060 25729 log.cc:824] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:33.312048 25729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: No bootstrap required, opened a new log
I20260806 16:12:33.315097 25729 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } }
I20260806 16:12:33.315330 25729 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:33.315364 25729 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fe45c2c8f4406a8cf84e1adc035c35, State: Initialized, Role: FOLLOWER
I20260806 16:12:33.315805 25729 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } }
I20260806 16:12:33.316313 25735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } } }
I20260806 16:12:33.316464 25735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:33.317065 25729 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:12:33.321868 25746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:12:33.321962 25746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:12:33.325611 25581 master_runner.cc:428] Detected that this master e5fe45c2c8f4406a8cf84e1adc035c35 is joining an existing cluster
I20260806 16:12:33.325672 25581 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.90.60:35739
I20260806 16:12:33.336746 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.328333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.18.90.62:32877)
0806 16:12:33.328472 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.328479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.328496 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:12:33.328545 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.328548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.328601 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.328681 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.329412 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.330255 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.331508 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.331860 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.331950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.333466 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.333470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.333476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.333506 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.335071 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.335502 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.335506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.335508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.335595 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.335879 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.335884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.335886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.336065 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.336172 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.336493 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.336606 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:12:33.336761 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.327786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:33888)
0806 16:12:33.328102 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.328121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.328222 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.328851 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.328855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.329126 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.329277 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.329284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.330385 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.330388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.331375 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.331385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.332111 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.332114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.332298 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.332890 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.332902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.333305 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.335200 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.335203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.335205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.335405 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.335693 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.335696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.335698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.335782 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.336161 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.336165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.336307 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.336427 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.336567 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260806 16:12:33.338419 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25066
I20260806 16:12:33.347354 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:45227
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:32877
--webserver_port=36337
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032751, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.62@KRBTEST.COM for oryx/127.18.90.60@KRBTEST.COM
I20260806 16:12:33.353539 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.343402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60956)
0806 16:12:33.343778 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.343805 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.343891 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.344439 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.344443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.344662 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.344803 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.344811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.345948 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.345955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.346926 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.346937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.347415 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.347419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.347515 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.347997 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.348011 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.349603 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.351803 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.351806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.351809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.352092 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.352483 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.352486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.352488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.352574 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.353021 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.353024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.353122 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.353202 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.353357 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260806 16:12:33.354542 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.343490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.18.90.60:35739)
0806 16:12:33.344116 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.344127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.344141 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:33.344212 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.344217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.344307 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.344398 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.344941 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.345823 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.347055 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.347298 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.347408 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.349774 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.349778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.349781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.349809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.351619 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.352271 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.352275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.352277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.352377 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.352679 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.352682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.352684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.352912 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.354142 (+ 1230us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.354235 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.354361 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"spinlock_wait_cycles":1408,"thread_start_us":128,"threads_started":1}
I20260806 16:12:33.355826 25003 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.90.60:35739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35"
member_type: NON_VOTER
last_known_addr {
host: "127.18.90.60"
port: 35739
}
attrs {
promote: true
}
}
dest_uuid: "249afc2e239449e1aad233ef197bd6c2"
cas_config_opid_index: 5
I20260806 16:12:33.356676 25003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } }
I20260806 16:12:33.357621 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.328684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.18.90.61:46489)
0806 16:12:33.328807 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.328812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.328863 (+ 51us) server_negotiation.cc:407] Connection header received
0806 16:12:33.329084 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.329087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.329138 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.329190 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.329932 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.330745 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.331963 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.332352 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.332431 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.350354 (+ 17923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.350367 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.350373 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.350412 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.352492 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.353420 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.353427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.353429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.353524 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.356427 (+ 2903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.356431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.356432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.356760 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.356842 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.357316 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.357465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:33.357683 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.328506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.18.90.60:35739)
0806 16:12:33.328610 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.328614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.329049 (+ 435us) server_negotiation.cc:407] Connection header received
0806 16:12:33.329195 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.329197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.329234 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.329279 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.329898 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.330580 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.331963 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.332362 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.332468 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.354133 (+ 21665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.354140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.354144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.354174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.355996 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.356469 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.356472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.356474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.356544 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.356807 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.356810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.356815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.357005 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.357078 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.357400 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.357518 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":40}
I20260806 16:12:33.358047 25569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.358609 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.327896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59138)
0806 16:12:33.328770 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.328784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.328933 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.329336 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.329338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.329612 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.329789 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.329797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.330889 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.330892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.331821 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.331831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.332581 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.332584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.332712 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.348778 (+ 16066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.348794 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.349225 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.352694 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.352697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.352698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.353275 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.353774 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.353777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.353779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.353878 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.357007 (+ 3129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.357010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.357126 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.358093 (+ 967us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.358422 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260806 16:12:33.359793 25569 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.90.60:35739
I20260806 16:12:33.360013 25569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.360136 25569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:33.362099 25702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:33.363395 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.358288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59152)
0806 16:12:33.358527 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.358544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.358619 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.359116 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.359119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.359139 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:33.359233 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.359239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.360412 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.360416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.361745 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.361756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.362967 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.362971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.363071 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.363119 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.363195 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.363270 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260806 16:12:33.363898 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.358754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.18.90.61:46489)
0806 16:12:33.358872 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.358876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.358888 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:33.358926 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.358929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.358985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.359080 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:33.359364 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.360177 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.361905 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.362623 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.362736 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.362903 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.363714 (+ 811us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.363787 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:12:33.363935 25702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Refusing update from remote peer 249afc2e239449e1aad233ef197bd6c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 16:12:33.364342 25581 master.cc:537] Master@127.18.90.60:35739 shutting down...
I20260806 16:12:33.364611 25619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 LEARNER]: Refusing update from remote peer 249afc2e239449e1aad233ef197bd6c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 16:12:33.365134 25567 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 16:12:33.367280 25619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.368831 25568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:33.370579 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249afc2e239449e1aad233ef197bd6c2. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.370711 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: LEARNER
I20260806 16:12:33.370803 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: NON_VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: true } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.370880 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: LEARNER
W20260806 16:12:33.371572 24986 consensus_peers.cc:487] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 -> Peer e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60:35739): Couldn't send request to peer e5fe45c2c8f4406a8cf84e1adc035c35. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 16:12:33.372745 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.328400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60952)
0806 16:12:33.328962 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.328979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.329105 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.329424 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.329427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.329654 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.329789 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.329797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.330684 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.330687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.331855 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.331864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.332579 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.332584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.332726 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.349767 (+ 17041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.349787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.353984 (+ 4197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.356111 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.356113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.356115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.356370 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.356635 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.356637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.356639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.356719 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.357138 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.357141 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.357219 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.372336 (+ 15117us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.372540 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":12,"thread_start_us":107,"threads_started":1}
I20260806 16:12:33.373678 25581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:33.373991 25581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:33.374128 25581 tablet_replica.cc:330] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: stopping tablet replica
I20260806 16:12:33.379536 25567 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: attempting to promote NON_VOTER a7348509d4614dfbb816db230c4229bb to VOTER
I20260806 16:12:33.380059 25567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } }
I20260806 16:12:33.380988 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.380598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60972)
0806 16:12:33.380845 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.380907 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 16:12:33.381389 25619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 LEARNER]: Refusing update from remote peer 249afc2e239449e1aad233ef197bd6c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 16:12:33.382370 25581 master.cc:552] Master@127.18.90.60:35739 shutdown complete.
I20260806 16:12:33.382457 25581 master_runner.cc:305] Clearing existing system tablet
I20260806 16:12:33.382510 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.382209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60980)
0806 16:12:33.382364 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.382435 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 16:12:33.382637 25568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 16:12:33.382900 25581 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.382994 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.384353 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.384035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60986)
0806 16:12:33.384158 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.384203 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 16:12:33.384482 24986 consensus_peers.cc:487] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 -> Peer e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60:35739): Couldn't send request to peer e5fe45c2c8f4406a8cf84e1adc035c35. Status: Network error: Client connection negotiation failed: client connection to 127.18.90.60:35739: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 16:12:33.385037 25581 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:33.385107 25569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a7348509d4614dfbb816db230c4229bb (127.18.90.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.385950 25760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.386183 25581 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.386296 25581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:33.386329 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
uuid: "e5fe45c2c8f4406a8cf84e1adc035c35"
format_stamp: "Formatted at 2026-08-06 16:12:33 on dist-test-slave-0b58"
server_key: "e53c4bbf0043a0966d68b2c5546ce130"
server_key_iv: "dff3dd82fa62ad35f8c119c54ed6ed42"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.387220 25619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a7348509d4614dfbb816db230c4229bb (127.18.90.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.388382 25581 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:33.390776 25581 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 16:12:33.390857 25581 log.cc:1196] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 16:12:33.390781 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.390895 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:33.390559 25569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.391054 25569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:33.391098 25581 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Deleting consensus metadata
I20260806 16:12:33.391265 25581 master_runner.cc:315] Copying system tablet from 127.18.90.62:32877
I20260806 16:12:33.392477 25758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.392608 25758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:33.392917 25581 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: tablet copy: Beginning tablet copy session from remote peer at address 127.18.90.62:32877
I20260806 16:12:33.402478 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.393513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.18.90.62:32877)
0806 16:12:33.393626 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.393631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.394101 (+ 470us) server_negotiation.cc:407] Connection header received
0806 16:12:33.394206 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.394209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.394261 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.394353 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.394867 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.395669 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.396860 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.397250 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.397383 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.398724 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.398728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.398731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.398758 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.400534 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.401031 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.401036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.401038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.401129 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.401467 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.401470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.401472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.401688 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.401759 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.402176 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.402318 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:12:33.403888 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.393719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:33892)
0806 16:12:33.393993 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.394009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.394125 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.394388 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.394391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.394606 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.394745 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.394752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.395786 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.395790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.396730 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.396741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.397397 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.397400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.397499 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.398110 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.398124 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.398571 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.400649 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.400652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.400655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.400924 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.401243 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.401246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.401248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.401357 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.401834 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.401837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.401936 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.403519 (+ 1583us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.403678 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":67,"threads_started":1}
I20260806 16:12:33.404425 25023 tablet_copy_service.cc:140] P 249afc2e239449e1aad233ef197bd6c2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e5fe45c2c8f4406a8cf84e1adc035c35 ({username='oryx', principal='oryx/127.18.90.60@KRBTEST.COM'} at 127.0.0.1:33892)
I20260806 16:12:33.404525 25023 tablet_copy_service.cc:161] P 249afc2e239449e1aad233ef197bd6c2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e5fe45c2c8f4406a8cf84e1adc035c35 at {username='oryx', principal='oryx/127.18.90.60@KRBTEST.COM'} at 127.0.0.1:33892: session id = e5fe45c2c8f4406a8cf84e1adc035c35-00000000000000000000000000000000
I20260806 16:12:33.405651 25023 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 16:12:33.407002 25581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 16:12:33.409618 25581 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: tablet copy: Starting download of 0 data blocks...
I20260806 16:12:33.409821 25581 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: tablet copy: Starting download of 1 WAL segments...
I20260806 16:12:33.411449 25581 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 16:12:33.413353 25023 tablet_copy_service.cc:342] P 249afc2e239449e1aad233ef197bd6c2: Request end of tablet copy session e5fe45c2c8f4406a8cf84e1adc035c35-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.90.60@KRBTEST.COM'} at 127.0.0.1:33892
I20260806 16:12:33.413444 25023 tablet_copy_service.cc:433] P 249afc2e239449e1aad233ef197bd6c2: ending tablet copy session e5fe45c2c8f4406a8cf84e1adc035c35-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e5fe45c2c8f4406a8cf84e1adc035c35
I20260806 16:12:33.420886 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:33.422501 25771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.422556 25768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.423215 25581 server_base.cc:1034] running on GCE node
W20260806 16:12:33.423316 25769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.423511 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:33.423844 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:33.424989 25581 hybrid_clock.cc:648] HybridClock initialized: now 1786032753424977 us; error 29 us; skew 500 ppm
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:33.426712 25581 init.cc:377] Logged in from keytab as oryx/127.18.90.60@KRBTEST.COM (short username oryx)
I20260806 16:12:33.427197 25581 webserver.cc:460] Webserver started at http://127.18.90.60:44951/ using document root <none> and password file <none>
I20260806 16:12:33.427394 25581 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.427438 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.428797 25581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:33.430016 25778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.430292 25581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:33.430369 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
uuid: "e5fe45c2c8f4406a8cf84e1adc035c35"
format_stamp: "Formatted at 2026-08-06 16:12:33 on dist-test-slave-0b58"
server_key: "e53c4bbf0043a0966d68b2c5546ce130"
server_key_iv: "dff3dd82fa62ad35f8c119c54ed6ed42"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.430454 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.453099 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:33.455739 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:33.461007 25581 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.60:35739
I20260806 16:12:33.461051 25830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.60:35739 every 8 connection(s)
I20260806 16:12:33.461617 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
I20260806 16:12:33.464465 25831 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:12:33.465576 25831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Bootstrap starting.
I20260806 16:12:33.477614 25831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:12:33.477958 25831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Bootstrap complete.
I20260806 16:12:33.478631 25831 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } }
I20260806 16:12:33.478783 25831 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5fe45c2c8f4406a8cf84e1adc035c35, State: Initialized, Role: LEARNER
I20260806 16:12:33.478900 25831 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } }
I20260806 16:12:33.479415 25831 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:33.479285 25834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.479677 25834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: This master's current role is: LEARNER
I20260806 16:12:33.484912 25845 catalog_manager.cc:1259] Loaded cluster ID: 39ab755326094acd89fc86cbad146351
I20260806 16:12:33.484984 25845 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: loading cluster ID for follower catalog manager: success
I20260806 16:12:33.487879 25845 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: acquiring CA information for follower catalog manager: success
I20260806 16:12:33.489379 25845 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 16:12:33.503448 25756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:33.503806 25756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:33.503890 25756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:12:33.506640 25331 heartbeater.cc:503] Master 127.18.90.62:32877 was elected leader, sending a full tablet report...
W20260806 16:12:33.508673 25756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:33.508754 25756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:33.508847 25756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:33.514532 25756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:45227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=36337
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:33.515705 25756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:33.516861 25756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:33.527026 25854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.527026 25852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.527050 25851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.527544 25756 server_base.cc:1034] running on GCE node
I20260806 16:12:33.527906 25756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:33.528569 25756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:33.529736 25756 hybrid_clock.cc:648] HybridClock initialized: now 1786032753529714 us; error 31 us; skew 500 ppm
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:33.532613 25756 init.cc:377] Logged in from keytab as oryx/127.18.90.1@KRBTEST.COM (short username oryx)
I20260806 16:12:33.533504 25756 webserver.cc:460] Webserver started at http://127.18.90.1:36337/ using document root <none> and password file <none>
I20260806 16:12:33.534082 25756 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.534224 25756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.538231 25756 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:12:33.540472 25862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.541745 25756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:33.541885 25756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "870db067f3704aceba4f9bae8a294d1c"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "a0940c67f0e71fdef0853952e8edd8d8"
server_key_iv: "4bc18ce92921f83f800e3245bf8403be"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.542464 25756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.561959 25756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:33.565222 25756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:33.565440 25756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:33.566073 25756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:33.566998 25756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:33.567070 25756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.567144 25756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:33.567193 25756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.576344 25756 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:45227
I20260806 16:12:33.576417 25975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:45227 every 8 connection(s)
I20260806 16:12:33.577459 25756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:33.587386 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25756
I20260806 16:12:33.587607 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25201
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.60@KRBTEST.COM
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
I20260806 16:12:33.594244 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.579971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:35463 (local address 127.18.90.61:46489)
0806 16:12:33.580107 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.580114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.580682 (+ 568us) server_negotiation.cc:407] Connection header received
0806 16:12:33.581856 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.581860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.581916 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.581972 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.583576 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.584431 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.585929 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.586293 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.586376 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.589888 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.589893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.589897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.589930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.591766 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592371 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.592377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.592381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.592472 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592879 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.592884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.592886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.593129 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.593210 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.593932 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.594062 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:33.594362 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.579886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:46597 (local address 127.18.90.60:35739)
0806 16:12:33.580159 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.580163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.581174 (+ 1011us) server_negotiation.cc:407] Connection header received
0806 16:12:33.581827 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.581830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.581894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.582007 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.583560 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.584321 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.585513 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.585764 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.585879 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.589436 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.589444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.589451 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.589489 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.591474 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592249 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.592290 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.592295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.592403 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592917 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.592924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.592928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.593194 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.593285 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.593934 (+ 649us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.594093 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260806 16:12:33.594489 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.579488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:41461 (local address 127.18.90.62:32877)
0806 16:12:33.579631 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.579636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.581120 (+ 1484us) server_negotiation.cc:407] Connection header received
0806 16:12:33.581857 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.581860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.581943 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.581996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.583576 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.584431 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.585930 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.586317 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.586385 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.590117 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.590123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.590128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.590639 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.592675 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.593149 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.593168 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.593171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.593261 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.593557 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.593560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.593561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.593768 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.593845 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.594192 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.594316 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:12:33.596129 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.580180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.18.90.1:46597)
0806 16:12:33.581053 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.581065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.581614 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.582236 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.582243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.582768 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.583411 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.583426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.584469 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.584474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.585381 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.585391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.585921 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.585925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.586105 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.586757 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.586773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.589231 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.591628 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.591638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.591660 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.592112 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.592540 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.592551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.592786 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.593363 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.593369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.593639 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.594529 (+ 890us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.595044 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":210,"spinlock_wait_cycles":8192,"thread_start_us":122,"threads_started":1}
I20260806 16:12:33.596124 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.579798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:41461)
0806 16:12:33.581002 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.581016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.581608 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.582238 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.582245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.582770 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.583413 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.583428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.584567 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.584570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.585787 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.585799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.586545 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.586548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.586688 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.587345 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.587363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.589675 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.592798 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.592808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.593036 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.593368 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.593371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.593373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.593456 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.593917 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.593920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.593997 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.594642 (+ 645us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.594939 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":439,"spinlock_wait_cycles":53888,"thread_start_us":168,"threads_started":1}
I20260806 16:12:33.596124 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.579799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.18.90.1:35463)
0806 16:12:33.580491 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.580562 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.581608 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.582238 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.582245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.582778 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.583413 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.583429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.584567 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.584570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.585792 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.585799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.586537 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.586542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.586691 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.587602 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.587618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.589728 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.591903 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.591907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.591911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.592208 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.592600 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.592604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.592606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.592766 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.593282 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.593291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.593639 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.595050 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.595170 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":331,"spinlock_wait_cycles":32640,"thread_start_us":734,"threads_started":1}
I20260806 16:12:33.597714 25978 heartbeater.cc:347] Connected to a master server at 127.18.90.61:46489
I20260806 16:12:33.597970 25976 heartbeater.cc:347] Connected to a master server at 127.18.90.60:35739
I20260806 16:12:33.597955 25978 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:33.598106 25976 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:33.598557 25976 heartbeater.cc:511] Master 127.18.90.60:35739 requested a full tablet report, sending...
I20260806 16:12:33.598565 25978 heartbeater.cc:511] Master 127.18.90.61:46489 requested a full tablet report, sending...
I20260806 16:12:33.599049 25977 heartbeater.cc:347] Connected to a master server at 127.18.90.62:32877
I20260806 16:12:33.599161 25977 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:33.599315 25977 heartbeater.cc:511] Master 127.18.90.62:32877 requested a full tablet report, sending...
I20260806 16:12:33.600127 25003 ts_manager.cc:194] Re-registered known tserver with Master: 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227)
I20260806 16:12:33.600154 25609 ts_manager.cc:194] Registered new tserver with Master: 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227)
I20260806 16:12:33.600383 25787 ts_manager.cc:194] Registered new tserver with Master: 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227)
I20260806 16:12:33.600914 25003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:41461
I20260806 16:12:33.602547 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:38549
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:32877
--webserver_port=36467
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:33.719767 25984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:33.720073 25984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:33.720158 25984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:12:33.721863 25466 heartbeater.cc:503] Master 127.18.90.62:32877 was elected leader, sending a full tablet report...
W20260806 16:12:33.724826 25984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:33.724910 25984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:33.724997 25984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:33.730510 25984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=36467
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:33.731729 25984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:33.732885 25984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:33.742810 25991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.742884 25984 server_base.cc:1034] running on GCE node
W20260806 16:12:33.742810 25990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.743023 25993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.743484 25984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:33.744127 25984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:33.745335 25984 hybrid_clock.cc:648] HybridClock initialized: now 1786032753745321 us; error 32 us; skew 500 ppm
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:33.748420 25984 init.cc:377] Logged in from keytab as oryx/127.18.90.2@KRBTEST.COM (short username oryx)
I20260806 16:12:33.749372 25984 webserver.cc:460] Webserver started at http://127.18.90.2:36467/ using document root <none> and password file <none>
I20260806 16:12:33.749949 25984 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.750030 25984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.754273 25984 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:12:33.756552 26000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.757858 25984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:12:33.758013 25984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "a5b3b108823940d2b7685eda5831fbab"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "50c38e6abf22e63b09e060422c899dd2"
server_key_iv: "4f17f52cf4654c46d296815a33a59083"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.758617 25984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.774617 25984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:33.777897 25984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:33.778113 25984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:33.778770 25984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:33.779820 25984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:33.779889 25984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.779963 25984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:33.780015 25984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.789189 25984 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38549
I20260806 16:12:33.789206 26113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38549 every 8 connection(s)
I20260806 16:12:33.790241 25984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:33.799760 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 25984
I20260806 16:12:33.799964 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25336
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.60@KRBTEST.COM
I20260806 16:12:33.807873 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.792523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:43649 (local address 127.18.90.61:46489)
0806 16:12:33.792664 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.792669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.793415 (+ 746us) server_negotiation.cc:407] Connection header received
0806 16:12:33.794498 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.794502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.794563 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.794665 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.796024 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.796643 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798306 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.798650 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.802905 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.802909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.802912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.802941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.805055 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.805822 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.805826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.805828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.805913 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.806387 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.806391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.806392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.806622 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.806693 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.807561 (+ 868us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.807699 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:12:33.807873 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.792137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:57971 (local address 127.18.90.62:32877)
0806 16:12:33.792308 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.792313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.793638 (+ 1325us) server_negotiation.cc:407] Connection header received
0806 16:12:33.794517 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.794520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.794578 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.794631 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.796039 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.796893 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798219 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.798563 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798663 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.802905 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.802909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.802912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.802941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.805177 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.805815 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.805819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.805820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.805913 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.806387 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.806391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.806392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.806622 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.806693 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.807562 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.807699 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:12:33.809762 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.792627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:57971)
0806 16:12:33.793544 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.793556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.794300 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.794850 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.794857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.795322 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.795871 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.795888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.797018 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.797021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.798089 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.798098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.798808 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.799085 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.800555 (+ 1470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.800569 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.802694 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.805300 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.805303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.805305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.805687 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.806033 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.806036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.806038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.806269 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.806783 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.806790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.807237 (+ 447us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.808048 (+ 811us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.808653 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":530,"spinlock_wait_cycles":42624,"thread_start_us":127,"threads_started":1}
I20260806 16:12:33.809767 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.792627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.18.90.2:43649)
0806 16:12:33.793201 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.793271 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.794284 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.794848 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.794855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.795314 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.795870 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.795884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.796788 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.796792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.798192 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.798202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.798805 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.799099 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.799773 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.799820 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.802723 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.805205 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.805215 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.805237 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.805687 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.806033 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.806036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.806038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.806266 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.806783 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.806790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.807237 (+ 447us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.808151 (+ 914us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.808573 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":211,"spinlock_wait_cycles":14464,"thread_start_us":703,"threads_started":1}
I20260806 16:12:33.810549 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.792204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:33605 (local address 127.18.90.60:35739)
0806 16:12:33.792366 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.792371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.793965 (+ 1594us) server_negotiation.cc:407] Connection header received
0806 16:12:33.794516 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.794520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.794578 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.794631 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:33.796041 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.796895 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798509 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.798828 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798944 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.806227 (+ 7283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:33.806234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:33.806240 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:33.806280 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:33.808223 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.808879 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.808885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.808890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.808984 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:33.809374 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:33.809377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:33.809379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:33.809623 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:33.809718 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.810175 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.810314 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 16:12:33.811244 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.793586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.18.90.2:33605)
0806 16:12:33.793855 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.793867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.794300 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.794850 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.794857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.795297 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:33.795871 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.795888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.797579 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.797583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.798397 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.798404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.798971 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.798975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.799115 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.803151 (+ 4036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:33.803171 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:33.806019 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:33.808394 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.808400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.808406 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.808754 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.809110 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:33.809114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:33.809117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:33.809222 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:33.809801 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:33.809807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:33.809925 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.810792 (+ 867us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.810979 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":127,"spinlock_wait_cycles":15744,"thread_start_us":105,"threads_started":1}
I20260806 16:12:33.811308 26115 heartbeater.cc:347] Connected to a master server at 127.18.90.62:32877
I20260806 16:12:33.811566 26115 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:33.811877 26114 heartbeater.cc:347] Connected to a master server at 127.18.90.60:35739
I20260806 16:12:33.811964 26114 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:33.812121 26115 heartbeater.cc:511] Master 127.18.90.62:32877 requested a full tablet report, sending...
I20260806 16:12:33.812140 26114 heartbeater.cc:511] Master 127.18.90.60:35739 requested a full tablet report, sending...
I20260806 16:12:33.812294 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:39809
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:32877
--webserver_port=44855
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:12:33.813230 25003 ts_manager.cc:194] Re-registered known tserver with Master: a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549)
I20260806 16:12:33.813359 25787 ts_manager.cc:194] Registered new tserver with Master: a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549)
I20260806 16:12:33.814033 25003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:57971
I20260806 16:12:33.814708 26116 heartbeater.cc:347] Connected to a master server at 127.18.90.61:46489
I20260806 16:12:33.814805 26116 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:33.815202 26116 heartbeater.cc:511] Master 127.18.90.61:46489 requested a full tablet report, sending...
I20260806 16:12:33.816026 25609 ts_manager.cc:194] Registered new tserver with Master: a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549)
I20260806 16:12:33.818816 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.814365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.18.90.60:35739)
0806 16:12:33.814471 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.814475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.814555 (+ 80us) server_negotiation.cc:407] Connection header received
0806 16:12:33.814710 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:33.814713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:33.814781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:33.814869 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:33.815289 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.815983 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.817711 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.818140 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.818223 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.818334 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:33.818561 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 16:12:33.818690 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 16:12:33.818822 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.814326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:60996)
0806 16:12:33.814497 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.814511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:33.814597 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:33.814991 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:33.814994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:33.815018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:33.815139 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.815148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.816298 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.816304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.817564 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:33.817579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:33.818272 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:33.818278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:33.818405 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:33.818444 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 16:12:33.818534 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:12:33.818638 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 16:12:33.819547 25797 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 16:12:33.819957 25834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249afc2e239449e1aad233ef197bd6c2. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:33.820101 25834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: This master's current role is: LEARNER
W20260806 16:12:33.929749 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:33.930024 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:33.930068 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:33.934335 26121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:33.934403 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:33.934466 26121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:33.940172 26121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:39809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=44855
--tserver_master_addrs=127.18.90.62:32877,127.18.90.61:46489,127.18.90.60:35739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:33.941382 26121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:33.942477 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:33.952780 26127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.952874 26121 server_base.cc:1034] running on GCE node
W20260806 16:12:33.952783 26126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:33.952783 26129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:33.953492 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:33.954130 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:33.955295 26121 hybrid_clock.cc:648] HybridClock initialized: now 1786032753955274 us; error 40 us; skew 500 ppm
Aug 06 16:12:33 dist-test-slave-0b58 krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:33.958405 26121 init.cc:377] Logged in from keytab as oryx/127.18.90.3@KRBTEST.COM (short username oryx)
I20260806 16:12:33.959326 26121 webserver.cc:460] Webserver started at http://127.18.90.3:44855/ using document root <none> and password file <none>
I20260806 16:12:33.959872 26121 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:33.959941 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:33.964025 26121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:33.966259 26136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.967561 26121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:12:33.967659 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "748c8722e114457bbb7fdd64d4548e0f"
format_stamp: "Formatted at 2026-08-06 16:12:32 on dist-test-slave-0b58"
server_key: "79a4e218a72749073ad0bf02201ff717"
server_key_iv: "b54170ede2246c8a002d14a6fe718bed"
server_key_version: "encryptionkey@0"
I20260806 16:12:33.968418 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:33.980063 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:33.983536 26121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:33.983740 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:33.984475 26121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:33.985358 26121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:33.985405 26121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.985445 26121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:33.985496 26121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:33.994632 26121 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:39809
I20260806 16:12:33.994660 26249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:39809 every 8 connection(s)
I20260806 16:12:33.995651 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:34.001582 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 26121
Aug 06 16:12:34 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.60@KRBTEST.COM
Aug 06 16:12:34 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.61@KRBTEST.COM
Aug 06 16:12:34 dist-test-slave-0b58 krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032753, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:34.014946 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.997635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:38893 (local address 127.18.90.60:35739)
0806 16:12:33.997904 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.997910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.998821 (+ 911us) server_negotiation.cc:407] Connection header received
0806 16:12:34.000520 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.000524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.000646 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.000758 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.002951 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.003673 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.005299 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.005646 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.005754 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.010297 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.010302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.010309 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.010338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.012210 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.012914 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.012920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.012923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.013015 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.013473 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.013482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.013485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.013757 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.013851 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.014600 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.014726 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260806 16:12:34.014958 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.997615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:56811 (local address 127.18.90.62:32877)
0806 16:12:33.997809 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.997813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.999104 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 16:12:34.000416 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.000422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.000486 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.000547 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.002515 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.003333 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.005408 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.005751 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.005845 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.010295 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.010299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.010302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.010331 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.012791 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.013337 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.013341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.013342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.013434 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.013792 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.013796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.013797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.014014 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.014096 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.014598 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.014760 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 16:12:34.016772 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.004161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.18.90.61:46489)
0806 16:12:34.006050 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.006055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.006068 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:34.006107 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.006110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.006180 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.006285 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.007402 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.008171 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.009885 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.010239 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.010341 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.012319 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.012324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.012329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.012357 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.014428 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.014906 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.014909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.014911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.015000 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.015434 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.015437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.015439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.015644 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.015768 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.016493 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.016620 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1810,"thread_start_us":104,"threads_started":1}
I20260806 16:12:34.016811 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.997963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:56811)
0806 16:12:33.998993 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.999004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.000133 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.000980 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.000989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.001564 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:34.002323 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.002345 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.003557 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.003560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.005266 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.005279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.006012 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.006015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.006168 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.007433 (+ 1265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:34.007448 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:34.010101 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:34.012931 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:34.012934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:34.012936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:34.013211 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:34.013560 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:34.013565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:34.013567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:34.013660 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:34.014154 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:34.014157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:34.014293 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.015195 (+ 902us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.015579 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":154,"spinlock_wait_cycles":114048,"thread_start_us":200,"threads_started":1}
I20260806 16:12:34.017515 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.997962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.18.90.3:38893)
0806 16:12:33.998658 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.998709 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.000148 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.001108 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.001114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.002684 (+ 1570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:34.002813 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.002821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.003855 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.003858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.005158 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.005167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.005926 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.005930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.006159 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.006920 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:34.006945 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:34.010123 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:34.012370 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:34.012377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:34.012395 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:34.012793 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:34.013138 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:34.013141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:34.013144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:34.013344 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:34.013930 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:34.013938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:34.014284 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.015587 (+ 1303us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.015751 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":343,"spinlock_wait_cycles":1792,"thread_start_us":188,"threads_started":1}
I20260806 16:12:34.018112 26251 heartbeater.cc:347] Connected to a master server at 127.18.90.62:32877
I20260806 16:12:34.018204 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.005496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.18.90.60:35739)
0806 16:12:34.008475 (+ 2979us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.008481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.008844 (+ 363us) server_negotiation.cc:407] Connection header received
0806 16:12:34.008894 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.008898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.008954 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.009037 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.009950 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.010715 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.011962 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.012387 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.012493 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.014113 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.014116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.014118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.014144 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.015862 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.016409 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.016412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.016413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.016498 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.016786 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.016790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.016793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.016995 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.017116 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.017785 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.018041 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2884,"thread_start_us":87,"threads_started":1}
I20260806 16:12:34.018364 26251 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:34.018890 26250 heartbeater.cc:347] Connected to a master server at 127.18.90.60:35739
I20260806 16:12:34.018975 26250 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:34.019003 26251 heartbeater.cc:511] Master 127.18.90.62:32877 requested a full tablet report, sending...
I20260806 16:12:34.019119 26250 heartbeater.cc:511] Master 127.18.90.60:35739 requested a full tablet report, sending...
I20260806 16:12:34.020191 25003 ts_manager.cc:194] Re-registered known tserver with Master: 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809)
I20260806 16:12:34.020493 25787 ts_manager.cc:194] Registered new tserver with Master: 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809)
I20260806 16:12:34.021026 25003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:56811
I20260806 16:12:34.023468 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.998290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:44227 (local address 127.18.90.61:46489)
0806 16:12:33.998414 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:33.998418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:33.999265 (+ 847us) server_negotiation.cc:407] Connection header received
0806 16:12:34.000443 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.000447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.000511 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.000570 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.002515 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.003359 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.006064 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.006447 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.006526 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.018470 (+ 11944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.018474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.018476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.018500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.021116 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.022166 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.022170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.022171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.022271 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.022579 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.022582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.022584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.022783 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.022864 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.023200 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.023339 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
I20260806 16:12:34.023990 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:33.998137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.18.90.3:44227)
0806 16:12:33.999155 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:33.999171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.000148 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.000979 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.000989 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.002143 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:34.002326 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.002346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.003558 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.003561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.005929 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.005936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.006666 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.006669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.006798 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.007212 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:34.007226 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:34.018324 (+ 11098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:34.021613 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:34.021619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:34.021624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:34.022045 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:34.022385 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:34.022388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:34.022390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:34.022474 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:34.022922 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:34.022925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:34.023021 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.023691 (+ 670us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.023858 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":164,"spinlock_wait_cycles":4608,"thread_start_us":736,"threads_started":1}
I20260806 16:12:34.024708 26252 heartbeater.cc:347] Connected to a master server at 127.18.90.61:46489
I20260806 16:12:34.024803 26252 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:34.024995 26252 heartbeater.cc:511] Master 127.18.90.61:46489 requested a full tablet report, sending...
I20260806 16:12:34.025712 25609 ts_manager.cc:194] Registered new tserver with Master: 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809)
I20260806 16:12:34.026000 26264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.004706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.18.90.62:32877)
0806 16:12:34.008118 (+ 3412us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.008123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.008136 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:34.008172 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.008176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.008225 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.009276 (+ 1051us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.009400 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.019104 (+ 9704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.020119 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.021028 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.021150 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.022649 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.022652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.022655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.022680 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.024221 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.024713 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.024718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.024719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.024812 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.025094 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.025098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.025100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.025290 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.025422 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.025666 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.025801 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3333,"thread_start_us":84,"threads_started":1}
I20260806 16:12:34.028769 25003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:34.031864 25003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:34.034471 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: NON_VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: true } } }
I20260806 16:12:34.034631 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:34.037825 25761 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: attempting to promote NON_VOTER e5fe45c2c8f4406a8cf84e1adc035c35 to VOTER
W20260806 16:12:34.038112 26270 consensus_queue.cc:1159] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a7348509d4614dfbb816db230c4229bb" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20260806 16:12:34.038473 25761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } }
I20260806 16:12:34.039834 25797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 LEARNER]: Refusing update from remote peer 249afc2e239449e1aad233ef197bd6c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 16:12:34.040225 26270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 16:12:34.040566 25619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Refusing update from remote peer 249afc2e239449e1aad233ef197bd6c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 16:12:34.041204 26270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 16:12:34.041967 26270 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.042791 25797 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.043215 25619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.043656 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.043751 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:34.043871 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.043929 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:34.044026 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.044085 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:34.045495 25834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.045610 25834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:34.045881 26278 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "249afc2e239449e1aad233ef197bd6c2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:34.046031 26278 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:34.054517 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.048849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.18.90.1:45227)
0806 16:12:34.049217 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.049246 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.049475 (+ 229us) server_negotiation.cc:407] Connection header received
0806 16:12:34.049564 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.049570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.049789 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.049945 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:34.050548 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.051385 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.053136 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.053903 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.054045 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.054187 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.054275 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.054369 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":78,"threads_started":1}
I20260806 16:12:34.054502 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.048468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45227 (local address 127.0.0.1:54778)
0806 16:12:34.049257 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:34.049275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.049373 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.050099 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.050101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.050116 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:34.050422 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.050432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.051520 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.051523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.052982 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.052996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.054048 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.054051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.054148 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.054171 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.054248 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.054323 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":307,"thread_start_us":95,"threads_started":1}
I20260806 16:12:34.054550 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.048789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:39809 (local address 127.0.0.1:35018)
0806 16:12:34.049382 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:34.049394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.049481 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.050090 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.050093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.050108 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:34.050229 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.050237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.051633 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.051637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.052810 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.052821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.054116 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.054120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.054218 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.054250 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.054315 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.054387 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":87,"thread_start_us":89,"threads_started":1}
I20260806 16:12:34.054919 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.048964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.18.90.3:39809)
0806 16:12:34.049289 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.049309 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.049490 (+ 181us) server_negotiation.cc:407] Connection header received
0806 16:12:34.049575 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.049602 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.049820 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.049954 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:34.050357 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.051504 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.052940 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.053787 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.053890 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.054051 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.054661 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.054776 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
I20260806 16:12:34.054673 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.048119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38549 (local address 127.0.0.1:43022)
0806 16:12:34.048287 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:34.048310 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.048395 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.049562 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.049565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.049586 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:34.049698 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.049704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.051603 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.051606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.053023 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.053032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.054298 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.054303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.054380 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.054397 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.054440 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.054491 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":151}
I20260806 16:12:34.056176 26279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.048337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.18.90.2:38549)
0806 16:12:34.048970 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.048984 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.049009 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:12:34.049096 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.049102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.049364 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.049860 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:34.050274 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.051497 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.053151 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.054154 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.054718 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.054856 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.055980 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.056067 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":114,"threads_started":1}
I20260806 16:12:34.058187 26184 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc027c5c4024cf98a313bd96d523870 (DEFAULT_TABLE table=test-table [id=d1525ca051794ad98ca987b3c1748a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:34.058187 25910 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc027c5c4024cf98a313bd96d523870 (DEFAULT_TABLE table=test-table [id=d1525ca051794ad98ca987b3c1748a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:34.058558 26048 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc027c5c4024cf98a313bd96d523870 (DEFAULT_TABLE table=test-table [id=d1525ca051794ad98ca987b3c1748a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:34.059363 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc027c5c4024cf98a313bd96d523870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:34.059363 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc027c5c4024cf98a313bd96d523870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:34.059378 26048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc027c5c4024cf98a313bd96d523870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:34.067022 26284 tablet_bootstrap.cc:492] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Bootstrap starting.
I20260806 16:12:34.067988 26286 tablet_bootstrap.cc:492] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Bootstrap starting.
I20260806 16:12:34.068617 26285 tablet_bootstrap.cc:492] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Bootstrap starting.
I20260806 16:12:34.069375 26284 tablet_bootstrap.cc:654] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:34.070123 26284 log.cc:824] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:34.070174 26286 tablet_bootstrap.cc:654] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:34.070637 26285 tablet_bootstrap.cc:654] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:34.071002 26286 log.cc:824] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:34.071455 26285 log.cc:824] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:34.072647 26284 tablet_bootstrap.cc:492] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: No bootstrap required, opened a new log
I20260806 16:12:34.072882 26284 ts_tablet_manager.cc:1397] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:34.073158 26286 tablet_bootstrap.cc:492] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: No bootstrap required, opened a new log
I20260806 16:12:34.073359 26286 ts_tablet_manager.cc:1397] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:34.073525 26285 tablet_bootstrap.cc:492] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: No bootstrap required, opened a new log
I20260806 16:12:34.073705 26285 ts_tablet_manager.cc:1397] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:12:34.076848 26284 raft_consensus.cc:348] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.077093 26284 raft_consensus.cc:374] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:34.077157 26284 raft_consensus.cc:729] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b3b108823940d2b7685eda5831fbab, State: Initialized, Role: FOLLOWER
I20260806 16:12:34.077616 26284 consensus_queue.cc:260] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.078063 26286 raft_consensus.cc:348] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.078333 26286 raft_consensus.cc:374] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:34.078397 26286 raft_consensus.cc:729] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748c8722e114457bbb7fdd64d4548e0f, State: Initialized, Role: FOLLOWER
I20260806 16:12:34.078217 26285 raft_consensus.cc:348] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.078456 26285 raft_consensus.cc:374] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:34.078498 26285 raft_consensus.cc:729] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870db067f3704aceba4f9bae8a294d1c, State: Initialized, Role: FOLLOWER
I20260806 16:12:34.078800 26284 ts_tablet_manager.cc:1428] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:12:34.078856 26286 consensus_queue.cc:260] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.078900 26285 consensus_queue.cc:260] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.079815 26285 ts_tablet_manager.cc:1428] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:12:34.079815 26286 ts_tablet_manager.cc:1428] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:12:34.080054 26251 heartbeater.cc:503] Master 127.18.90.62:32877 was elected leader, sending a full tablet report...
W20260806 16:12:34.080024 25979 tablet.cc:2369] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:34.081799 26115 heartbeater.cc:503] Master 127.18.90.62:32877 was elected leader, sending a full tablet report...
I20260806 16:12:34.083017 25977 heartbeater.cc:503] Master 127.18.90.62:32877 was elected leader, sending a full tablet report...
I20260806 16:12:34.156188 26291 raft_consensus.cc:482] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:34.156437 26291 raft_consensus.cc:504] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.157647 26291 leader_election.cc:288] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227), a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549)
I20260806 16:12:34.163046 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.158083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:39503 (local address 127.18.90.1:45227)
0806 16:12:34.158249 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.158253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.158268 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:34.158317 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.158320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.158368 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.158455 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:34.158946 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.159750 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.161647 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.162308 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.162418 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.162485 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.162829 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.162907 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:34.163046 26279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.158097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:54727 (local address 127.18.90.2:38549)
0806 16:12:34.158221 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.158227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.158245 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:12:34.158311 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.158315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.158368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.158456 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:34.158946 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.159750 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.161647 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.162308 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.162432 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.162485 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.162829 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.162907 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:34.163108 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.157962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38549 (local address 127.18.90.3:54727)
0806 16:12:34.158122 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:34.158134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.158218 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.158485 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.158488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.158521 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:34.158824 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.158829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.159868 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.159871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.161514 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.161525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.162434 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.162438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.162657 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.162713 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.162795 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.162929 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":388,"spinlock_wait_cycles":7168}
I20260806 16:12:34.163323 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.157850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45227 (local address 127.18.90.3:39503)
0806 16:12:34.157998 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:34.158015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:34.158112 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:34.158485 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:34.158488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:34.158523 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:34.158821 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.158828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.159868 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.159871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.161514 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:34.161525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.162434 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.162438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:34.162660 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.162716 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 16:12:34.162795 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:12:34.162929 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":382,"spinlock_wait_cycles":9856}
I20260806 16:12:34.163849 26068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc027c5c4024cf98a313bd96d523870" candidate_uuid: "748c8722e114457bbb7fdd64d4548e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b3b108823940d2b7685eda5831fbab" is_pre_election: true
I20260806 16:12:34.164098 25930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc027c5c4024cf98a313bd96d523870" candidate_uuid: "748c8722e114457bbb7fdd64d4548e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870db067f3704aceba4f9bae8a294d1c" is_pre_election: true
I20260806 16:12:34.164214 26068 raft_consensus.cc:2454] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748c8722e114457bbb7fdd64d4548e0f in term 0.
I20260806 16:12:34.164404 25930 raft_consensus.cc:2454] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748c8722e114457bbb7fdd64d4548e0f in term 0.
I20260806 16:12:34.164800 26138 leader_election.cc:302] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748c8722e114457bbb7fdd64d4548e0f, a5b3b108823940d2b7685eda5831fbab; no voters:
I20260806 16:12:34.165138 26291 raft_consensus.cc:2781] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:34.165226 26291 raft_consensus.cc:482] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:34.165258 26291 raft_consensus.cc:3037] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:34.166401 26291 raft_consensus.cc:504] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.166765 26291 leader_election.cc:288] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [CANDIDATE]: Term 1 election: Requested vote from peers 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227), a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549)
I20260806 16:12:34.167119 25930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc027c5c4024cf98a313bd96d523870" candidate_uuid: "748c8722e114457bbb7fdd64d4548e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870db067f3704aceba4f9bae8a294d1c"
I20260806 16:12:34.167119 26068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc027c5c4024cf98a313bd96d523870" candidate_uuid: "748c8722e114457bbb7fdd64d4548e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b3b108823940d2b7685eda5831fbab"
I20260806 16:12:34.167249 25930 raft_consensus.cc:3037] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:34.167249 26068 raft_consensus.cc:3037] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:34.168588 25930 raft_consensus.cc:2454] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748c8722e114457bbb7fdd64d4548e0f in term 1.
I20260806 16:12:34.168588 26068 raft_consensus.cc:2454] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748c8722e114457bbb7fdd64d4548e0f in term 1.
I20260806 16:12:34.168916 26138 leader_election.cc:302] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748c8722e114457bbb7fdd64d4548e0f, a5b3b108823940d2b7685eda5831fbab; no voters:
I20260806 16:12:34.169093 26291 raft_consensus.cc:2781] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:34.169366 26291 raft_consensus.cc:686] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 1 LEADER]: Becoming Leader. State: Replica: 748c8722e114457bbb7fdd64d4548e0f, State: Running, Role: LEADER
I20260806 16:12:34.169984 26291 consensus_queue.cc:237] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } }
I20260806 16:12:34.173489 25003 catalog_manager.cc:5495] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3). New cstate: current_term: 1 leader_uuid: "748c8722e114457bbb7fdd64d4548e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:34.208031 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.204559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.18.90.3:39809)
0806 16:12:34.204698 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.204702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.204724 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:12:34.204802 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.204805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.204863 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.204954 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:34.205196 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.205717 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.206553 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.206757 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.206833 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.207169 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:34.207657 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:34.207780 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.207838 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.207899 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:12:34.214236 26068 raft_consensus.cc:1261] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Refusing update from remote peer 748c8722e114457bbb7fdd64d4548e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:34.214229 25930 raft_consensus.cc:1261] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 1 FOLLOWER]: Refusing update from remote peer 748c8722e114457bbb7fdd64d4548e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:34.215083 26291 consensus_queue.cc:1035] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:34.215312 26293 consensus_queue.cc:1035] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:34.226254 26297 mvcc.cc:205] Tried to move back new op lower bound from 7315590161254350848 to 7315590161083027456. Current Snapshot: MvccSnapshot[applied={T|T < 7315590161254350848}]
I20260806 16:12:34.228087 26298 mvcc.cc:205] Tried to move back new op lower bound from 7315590161254350848 to 7315590161083027456. Current Snapshot: MvccSnapshot[applied={T|T < 7315590161254350848}]
I20260806 16:12:34.239892 25002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:34.240084 25002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:34.244535 25002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 249afc2e239449e1aad233ef197bd6c2: Sending DeleteTablet for 3 replicas of tablet 2cc027c5c4024cf98a313bd96d523870
I20260806 16:12:34.245375 26184 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc027c5c4024cf98a313bd96d523870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:34 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:35018
I20260806 16:12:34.245569 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc027c5c4024cf98a313bd96d523870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:34 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:54778
I20260806 16:12:34.246068 26048 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc027c5c4024cf98a313bd96d523870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:34 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:43022
I20260806 16:12:34.246501 26307 tablet_replica.cc:330] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: stopping tablet replica
I20260806 16:12:34.246860 26307 raft_consensus.cc:2229] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:34.247216 26307 raft_consensus.cc:2258] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:34.247596 26306 tablet_replica.cc:330] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: stopping tablet replica
I20260806 16:12:34.247911 26306 raft_consensus.cc:2229] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:34.248132 26308 tablet_replica.cc:330] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: stopping tablet replica
I20260806 16:12:34.248334 26306 raft_consensus.cc:2258] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:34.248529 26308 raft_consensus.cc:2229] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:34.249059 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 24970
I20260806 16:12:34.250990 26306 ts_tablet_manager.cc:1905] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:34.251022 26307 ts_tablet_manager.cc:1905] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:34.253788 26306 ts_tablet_manager.cc:1918] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:34.253890 26306 log.cc:1196] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/2cc027c5c4024cf98a313bd96d523870
I20260806 16:12:34.254212 26306 ts_tablet_manager.cc:1939] T 2cc027c5c4024cf98a313bd96d523870 P 748c8722e114457bbb7fdd64d4548e0f: Deleting consensus metadata
I20260806 16:12:34.254807 26307 ts_tablet_manager.cc:1918] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:34.254969 26307 log.cc:1196] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/2cc027c5c4024cf98a313bd96d523870
I20260806 16:12:34.255379 26307 ts_tablet_manager.cc:1939] T 2cc027c5c4024cf98a313bd96d523870 P 870db067f3704aceba4f9bae8a294d1c: Deleting consensus metadata
I20260806 16:12:34.255466 26308 raft_consensus.cc:2258] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:34.257146 26308 ts_tablet_manager.cc:1905] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 16:12:34.258822 26138 connection.cc:535] server connection from 127.0.0.1:35018 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 16:12:34.258857 25863 connection.cc:535] server connection from 127.0.0.1:54778 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:12:34.259770 26308 ts_tablet_manager.cc:1918] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:34.259864 26308 log.cc:1196] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/2cc027c5c4024cf98a313bd96d523870
I20260806 16:12:34.260179 26308 ts_tablet_manager.cc:1939] T 2cc027c5c4024cf98a313bd96d523870 P a5b3b108823940d2b7685eda5831fbab: Deleting consensus metadata
W20260806 16:12:34.261699 26003 connection.cc:441] server connection from 127.0.0.1:43022 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43022 (request call id 1) could send its response
I20260806 16:12:34.271543 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.247621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.18.90.61:46489)
0806 16:12:34.247745 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.247750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.260587 (+ 12837us) server_negotiation.cc:407] Connection header received
0806 16:12:34.260698 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.260701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.260804 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.260888 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.262613 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.263382 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.264658 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.264977 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.265072 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.266860 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.266863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.266866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.266890 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.269201 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.269699 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.269703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.269708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.269805 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.270141 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.270145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.270147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.270422 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.270564 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.271238 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.271369 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:12:34.274497 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.263452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.18.90.61:46489)
0806 16:12:34.263571 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.263575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.263587 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:34.263657 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.263660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.263705 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.263785 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.264735 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.265433 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.266784 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.267150 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.267227 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.269219 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.269223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.269225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.269252 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.271579 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.273098 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.273102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.273104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.273216 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.273559 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.273562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.273564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.273785 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.273905 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.274222 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.274341 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:12:34.274710 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.263008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.18.90.60:35739)
0806 16:12:34.263129 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.263134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.264126 (+ 992us) server_negotiation.cc:407] Connection header received
0806 16:12:34.264357 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.264363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.264418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.264470 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.265449 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.266268 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.267480 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.267831 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.267907 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.269547 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.269550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.269553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.269580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.271368 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.273393 (+ 2025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.273399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.273402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.273484 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.273750 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.273753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.273755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.273957 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.274088 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.274430 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.274558 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:12:34.280115 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:34.247771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.18.90.60:35739)
0806 16:12:34.247873 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:34.247877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:34.271474 (+ 23597us) server_negotiation.cc:407] Connection header received
0806 16:12:34.271509 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:34.271512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:34.271563 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:34.271633 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:34.272460 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.273274 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.274533 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:34.274896 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:34.274970 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:34.276869 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:34.276872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:34.276875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:34.276898 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:34.278315 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.278780 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.278783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.278785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.278863 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:34.279136 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:34.279139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:34.279141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:34.279312 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:34.279485 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:34.279819 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 16:12:34.279942 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 16:12:35.231391 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.230772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:34111)
0806 16:12:35.231126 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.231234 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":87,"threads_started":1}
W20260806 16:12:35.231745 25977 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:32877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.233913 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.233507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:38463)
0806 16:12:35.233738 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.233818 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
W20260806 16:12:35.234256 26251 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:32877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.234690 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.234284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:50259)
0806 16:12:35.234542 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.234612 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
W20260806 16:12:35.234972 26115 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:32877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.745209 26319 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 249afc2e239449e1aad233ef197bd6c2)
I20260806 16:12:35.745262 26320 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 249afc2e239449e1aad233ef197bd6c2)
I20260806 16:12:35.745445 26319 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } }
I20260806 16:12:35.745476 26320 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } }
I20260806 16:12:35.746577 26319 leader_election.cc:288] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877), a7348509d4614dfbb816db230c4229bb (127.18.90.61:46489)
I20260806 16:12:35.746973 26320 leader_election.cc:288] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877), e5fe45c2c8f4406a8cf84e1adc035c35 (127.18.90.60:35739)
I20260806 16:12:35.747349 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.746775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34390)
0806 16:12:35.747153 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.747231 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":68,"threads_started":1}
W20260806 16:12:35.747501 25781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.748019 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.747243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34400)
0806 16:12:35.747847 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.747924 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":72,"threads_started":1}
W20260806 16:12:35.748327 25595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.748458 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.748187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34416)
0806 16:12:35.748317 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.748373 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 16:12:35.748621 25781 leader_election.cc:334] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877): Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.749454 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.749213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34428)
0806 16:12:35.749332 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.749387 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 16:12:35.749655 25595 leader_election.cc:334] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877): Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.751224 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.746797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:46489 (local address 127.0.0.1:59196)
0806 16:12:35.747223 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.747239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.747351 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.747765 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.747768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.747782 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.747912 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.747921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.748927 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.748930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.750123 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.750134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.750787 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.750791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.750894 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.750925 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.751000 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.751128 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":57,"thread_start_us":66,"threads_started":1}
I20260806 16:12:35.751278 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.747006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.18.90.61:46489)
0806 16:12:35.747408 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.747415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.747434 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:12:35.747485 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.747487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.747539 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.747630 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.748033 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.748807 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.750256 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.750665 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.750748 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.750796 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.751044 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.751132 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":326,"thread_start_us":56,"threads_started":1}
I20260806 16:12:35.751843 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7348509d4614dfbb816db230c4229bb" is_pre_election: true
I20260806 16:12:35.752090 25619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5fe45c2c8f4406a8cf84e1adc035c35 in term 1.
I20260806 16:12:35.752457 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.746785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:35739 (local address 127.0.0.1:32798)
0806 16:12:35.747543 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.747556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.747638 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.747941 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.747947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.747969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.748799 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.748805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.749794 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.749797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.750793 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.750803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.751960 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.751963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.752077 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.752101 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.752200 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.752322 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":69,"threads_started":1}
I20260806 16:12:35.752487 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.747187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.18.90.60:35739)
0806 16:12:35.747667 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.747674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.747691 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:12:35.747736 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.747741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.747783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.747855 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.748970 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.749671 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.750929 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.751849 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.751971 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.752026 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.752231 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.752332 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":367,"thread_start_us":49,"threads_started":1}
I20260806 16:12:35.752502 25782 leader_election.cc:302] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7348509d4614dfbb816db230c4229bb, e5fe45c2c8f4406a8cf84e1adc035c35; no voters: 249afc2e239449e1aad233ef197bd6c2
I20260806 16:12:35.752851 26319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 16:12:35.752920 26319 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 249afc2e239449e1aad233ef197bd6c2)
I20260806 16:12:35.752951 26319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:12:35.752950 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7348509d4614dfbb816db230c4229bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" is_pre_election: true
I20260806 16:12:35.754040 26319 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } }
I20260806 16:12:35.754315 25797 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a7348509d4614dfbb816db230c4229bb in current term 2: Already voted for candidate e5fe45c2c8f4406a8cf84e1adc035c35 in this term.
I20260806 16:12:35.754465 26319 leader_election.cc:288] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [CANDIDATE]: Term 2 election: Requested vote from peers 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877), a7348509d4614dfbb816db230c4229bb (127.18.90.61:46489)
I20260806 16:12:35.754806 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.754555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34430)
0806 16:12:35.754672 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.754731 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 16:12:35.754865 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7348509d4614dfbb816db230c4229bb"
I20260806 16:12:35.754987 25619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:12:35.755223 25593 leader_election.cc:302] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7348509d4614dfbb816db230c4229bb; no voters: 249afc2e239449e1aad233ef197bd6c2, e5fe45c2c8f4406a8cf84e1adc035c35
I20260806 16:12:35.756057 25619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5fe45c2c8f4406a8cf84e1adc035c35 in term 2.
I20260806 16:12:35.756201 26320 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:12:35.756424 25782 leader_election.cc:302] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7348509d4614dfbb816db230c4229bb, e5fe45c2c8f4406a8cf84e1adc035c35; no voters:
I20260806 16:12:35.756654 26319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:12:35.756866 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.756650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34438)
0806 16:12:35.756769 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.756820 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 16:12:35.756914 26319 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [term 2 LEADER]: Becoming Leader. State: Replica: e5fe45c2c8f4406a8cf84e1adc035c35, State: Running, Role: LEADER
W20260806 16:12:35.756981 25781 leader_election.cc:334] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877): Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
I20260806 16:12:35.757169 26319 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } }
I20260806 16:12:35.758356 26329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5fe45c2c8f4406a8cf84e1adc035c35. Latest consensus state: current_term: 2 leader_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:35.758451 26329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:35.758772 26331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:35.761106 26331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:35.761535 26331 catalog_manager.cc:1259] Loaded cluster ID: 39ab755326094acd89fc86cbad146351
I20260806 16:12:35.761616 26331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:35.762192 26331 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:35.762526 26331 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Loaded TSK: 0
I20260806 16:12:35.763290 26331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:12:35.797607 25787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:35.800890 25787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:35.804278 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.803920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34462)
0806 16:12:35.804052 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.804137 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 16:12:35.804351 25619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [term 2 FOLLOWER]: Refusing update from remote peer e5fe45c2c8f4406a8cf84e1adc035c35: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 16:12:35.805030 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.804830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.0.0.1:34466)
0806 16:12:35.804925 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.804973 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 16:12:35.804936 26329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 16:12:35.805295 25781 consensus_peers.cc:487] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 -> Peer 249afc2e239449e1aad233ef197bd6c2 (127.18.90.62:32877): Couldn't send request to peer 249afc2e239449e1aad233ef197bd6c2. Status: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 16:12:35.806636 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5fe45c2c8f4406a8cf84e1adc035c35. Latest consensus state: current_term: 2 leader_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:35.806777 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:35.807888 26329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:35.808030 26329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:35.808738 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "249afc2e239449e1aad233ef197bd6c2" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32877 } } peers { permanent_uuid: "a7348509d4614dfbb816db230c4229bb" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 46489 } attrs { promote: false } } peers { permanent_uuid: "e5fe45c2c8f4406a8cf84e1adc035c35" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 35739 } attrs { promote: false } } }
I20260806 16:12:35.808881 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7348509d4614dfbb816db230c4229bb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:12:35.818646 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.814209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38549 (local address 127.0.0.1:43030)
0806 16:12:35.814360 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.814377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.814460 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.815220 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.815223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.815240 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.815348 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.815355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.816361 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.816365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.817512 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.817521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.818227 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.818231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.818326 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.818353 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.818405 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.818496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":20}
I20260806 16:12:35.819176 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.814634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.18.90.2:38549)
0806 16:12:35.814920 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.814945 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.814963 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:12:35.815021 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.815024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.815079 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.815212 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.815490 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.816214 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.817631 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.818090 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.818903 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.818978 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.819009 (+ 31us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.819072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260806 16:12:35.819705 26048 tablet_service.cc:1467] Processing CreateTablet for tablet 5f745d2dee3d463e9f699080b30f99e5 (DEFAULT_TABLE table=test-table [id=6552fe5b152b42f0837d3a3889c64e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:35.819762 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.814590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:39809 (local address 127.0.0.1:35046)
0806 16:12:35.814880 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.814895 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.814975 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.816118 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.816121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.816138 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.816234 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.816240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.817266 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.817270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.818456 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.818467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.819404 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.819408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.819499 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.819524 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.819571 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.819640 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":51}
I20260806 16:12:35.820025 26048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f745d2dee3d463e9f699080b30f99e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:35.820353 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.814683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.18.90.3:39809)
0806 16:12:35.815883 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.815888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.815904 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:12:35.815959 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.815965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.816011 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.816112 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.816386 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.817166 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.818599 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.819251 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.819967 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.820040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.820080 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.820169 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":69,"threads_started":1}
I20260806 16:12:35.820917 26184 tablet_service.cc:1467] Processing CreateTablet for tablet 5f745d2dee3d463e9f699080b30f99e5 (DEFAULT_TABLE table=test-table [id=6552fe5b152b42f0837d3a3889c64e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:35.821012 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.815381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45227 (local address 127.0.0.1:54780)
0806 16:12:35.815801 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.815815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.815901 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.816319 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.816324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.816348 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.816445 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.816450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.817188 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.817193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.818941 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.818952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.820390 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.820397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.820716 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.820740 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.820819 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.820891 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":66,"thread_start_us":141,"threads_started":1}
I20260806 16:12:35.821213 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f745d2dee3d463e9f699080b30f99e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:35.821319 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.815535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.18.90.1:45227)
0806 16:12:35.815963 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.815967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.815981 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:35.816033 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.816036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.816078 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.816159 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.816565 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.817069 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.819087 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.820138 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.820237 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.820327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.820863 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.820958 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":77,"threads_started":1}
I20260806 16:12:35.822728 26341 tablet_bootstrap.cc:492] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Bootstrap starting.
I20260806 16:12:35.823235 25910 tablet_service.cc:1467] Processing CreateTablet for tablet 5f745d2dee3d463e9f699080b30f99e5 (DEFAULT_TABLE table=test-table [id=6552fe5b152b42f0837d3a3889c64e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:35.823529 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f745d2dee3d463e9f699080b30f99e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:35.823812 26342 tablet_bootstrap.cc:492] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Bootstrap starting.
I20260806 16:12:35.823859 26341 tablet_bootstrap.cc:654] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:35.824844 26342 tablet_bootstrap.cc:654] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:35.825451 26341 tablet_bootstrap.cc:492] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: No bootstrap required, opened a new log
I20260806 16:12:35.825655 26341 ts_tablet_manager.cc:1397] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:35.826324 26341 raft_consensus.cc:348] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.826406 26345 tablet_bootstrap.cc:492] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c: Bootstrap starting.
I20260806 16:12:35.826463 26342 tablet_bootstrap.cc:492] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: No bootstrap required, opened a new log
I20260806 16:12:35.826462 26341 raft_consensus.cc:374] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:35.826504 26341 raft_consensus.cc:729] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b3b108823940d2b7685eda5831fbab, State: Initialized, Role: FOLLOWER
I20260806 16:12:35.826551 26342 ts_tablet_manager.cc:1397] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:35.826665 26341 consensus_queue.cc:260] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.827015 26341 ts_tablet_manager.cc:1428] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:35.827021 26342 raft_consensus.cc:348] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.827142 26342 raft_consensus.cc:374] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:35.827180 26342 raft_consensus.cc:729] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748c8722e114457bbb7fdd64d4548e0f, State: Initialized, Role: FOLLOWER
I20260806 16:12:35.827330 26342 consensus_queue.cc:260] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.827445 26345 tablet_bootstrap.cc:654] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:35.827720 26342 ts_tablet_manager.cc:1428] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:12:35.828837 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.828394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:45615)
0806 16:12:35.828669 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.828737 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260806 16:12:35.829886 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.829542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:43147)
0806 16:12:35.829777 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.829822 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260806 16:12:35.829906 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.827507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:52701)
0806 16:12:35.829739 (+ 2232us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.829801 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2152,"thread_start_us":89,"threads_started":1}
I20260806 16:12:35.830405 26250 heartbeater.cc:503] Master 127.18.90.60:35739 was elected leader, sending a full tablet report...
I20260806 16:12:35.830956 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.830591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:34681)
0806 16:12:35.830848 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.830896 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260806 16:12:35.831086 26114 heartbeater.cc:503] Master 127.18.90.60:35739 was elected leader, sending a full tablet report...
I20260806 16:12:35.832211 26345 tablet_bootstrap.cc:492] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c: No bootstrap required, opened a new log
I20260806 16:12:35.832356 26345 ts_tablet_manager.cc:1397] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 16:12:35.832794 26345 raft_consensus.cc:348] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.832917 26345 raft_consensus.cc:374] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:35.832963 26345 raft_consensus.cc:729] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870db067f3704aceba4f9bae8a294d1c, State: Initialized, Role: FOLLOWER
I20260806 16:12:35.833103 26345 consensus_queue.cc:260] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.833606 26345 ts_tablet_manager.cc:1428] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:12:35.834251 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.833815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:42865)
0806 16:12:35.834070 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.834116 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260806 16:12:35.835103 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.834664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:58675)
0806 16:12:35.834936 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.834998 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
I20260806 16:12:35.835124 25976 heartbeater.cc:503] Master 127.18.90.60:35739 was elected leader, sending a full tablet report...
I20260806 16:12:35.867077 26351 raft_consensus.cc:482] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:35.867231 26351 raft_consensus.cc:504] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.868463 26351 leader_election.cc:288] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549), 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809)
I20260806 16:12:35.872522 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.868693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38549 (local address 127.18.90.1:32899)
0806 16:12:35.869049 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.869065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.869177 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.869469 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.869472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.869486 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.869603 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.869611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.870337 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.870342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.871266 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.871276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.872017 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.872022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.872165 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.872183 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.872315 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.872382 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":50}
I20260806 16:12:35.872519 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.868863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:43165 (local address 127.18.90.3:39809)
0806 16:12:35.868978 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.868981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.869049 (+ 68us) server_negotiation.cc:407] Connection header received
0806 16:12:35.869179 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.869183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.869242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.869323 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.869647 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.870216 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.871366 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.871794 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.871877 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.871978 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.872287 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.872378 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 16:12:35.872548 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.868810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:32899 (local address 127.18.90.2:38549)
0806 16:12:35.868928 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.868933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.869161 (+ 228us) server_negotiation.cc:407] Connection header received
0806 16:12:35.869274 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.869276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.869352 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.869433 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:35.869720 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.870211 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.871429 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.871816 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.871972 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.872041 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.872345 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.872434 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 16:12:35.872548 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.868732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:39809 (local address 127.18.90.1:43165)
0806 16:12:35.868940 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.868957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:35.869075 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:35.869350 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:35.869353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:35.869372 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:35.869523 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.869530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.870371 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.870374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.871238 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:35.871247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.871997 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.872001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:35.872133 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.872158 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:12:35.872221 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 16:12:35.872349 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":9}
I20260806 16:12:35.873164 26068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f745d2dee3d463e9f699080b30f99e5" candidate_uuid: "870db067f3704aceba4f9bae8a294d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b3b108823940d2b7685eda5831fbab" is_pre_election: true
I20260806 16:12:35.873340 26068 raft_consensus.cc:2454] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870db067f3704aceba4f9bae8a294d1c in term 0.
I20260806 16:12:35.873317 26204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f745d2dee3d463e9f699080b30f99e5" candidate_uuid: "870db067f3704aceba4f9bae8a294d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748c8722e114457bbb7fdd64d4548e0f" is_pre_election: true
I20260806 16:12:35.873567 26204 raft_consensus.cc:2454] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870db067f3704aceba4f9bae8a294d1c in term 0.
I20260806 16:12:35.873765 25864 leader_election.cc:302] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870db067f3704aceba4f9bae8a294d1c, a5b3b108823940d2b7685eda5831fbab; no voters:
I20260806 16:12:35.874086 26351 raft_consensus.cc:2781] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:35.874176 26351 raft_consensus.cc:482] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:35.874231 26351 raft_consensus.cc:3037] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:35.875563 26351 raft_consensus.cc:504] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.876085 26351 leader_election.cc:288] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [CANDIDATE]: Term 1 election: Requested vote from peers a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549), 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809)
I20260806 16:12:35.876485 26068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f745d2dee3d463e9f699080b30f99e5" candidate_uuid: "870db067f3704aceba4f9bae8a294d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b3b108823940d2b7685eda5831fbab"
I20260806 16:12:35.876503 26204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f745d2dee3d463e9f699080b30f99e5" candidate_uuid: "870db067f3704aceba4f9bae8a294d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748c8722e114457bbb7fdd64d4548e0f"
I20260806 16:12:35.876598 26068 raft_consensus.cc:3037] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:35.876638 26204 raft_consensus.cc:3037] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:35.877561 26068 raft_consensus.cc:2454] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870db067f3704aceba4f9bae8a294d1c in term 1.
I20260806 16:12:35.877602 26204 raft_consensus.cc:2454] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870db067f3704aceba4f9bae8a294d1c in term 1.
I20260806 16:12:35.877882 25864 leader_election.cc:302] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870db067f3704aceba4f9bae8a294d1c, a5b3b108823940d2b7685eda5831fbab; no voters:
I20260806 16:12:35.878108 26351 raft_consensus.cc:2781] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:35.878371 26351 raft_consensus.cc:686] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [term 1 LEADER]: Becoming Leader. State: Replica: 870db067f3704aceba4f9bae8a294d1c, State: Running, Role: LEADER
I20260806 16:12:35.878894 26351 consensus_queue.cc:237] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } }
I20260806 16:12:35.879180 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.878863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:59735)
0806 16:12:35.878979 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.879091 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 16:12:35.881596 25787 catalog_manager.cc:5495] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 870db067f3704aceba4f9bae8a294d1c (127.18.90.1). New cstate: current_term: 1 leader_uuid: "870db067f3704aceba4f9bae8a294d1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870db067f3704aceba4f9bae8a294d1c" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45227 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:35.899123 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.895710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.18.90.1:45227)
0806 16:12:35.895836 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:35.895846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:35.895860 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:35.895968 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:35.895970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:35.896038 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:35.896137 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:35.896452 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.896969 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.897768 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:35.897983 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:35.898060 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:35.898315 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:35.898787 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:35.898886 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:35.898952 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 16:12:35.899007 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:12:35.902238 26068 raft_consensus.cc:1261] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Refusing update from remote peer 870db067f3704aceba4f9bae8a294d1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:35.902467 26204 raft_consensus.cc:1261] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Refusing update from remote peer 870db067f3704aceba4f9bae8a294d1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:35.902848 26354 consensus_queue.cc:1035] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b3b108823940d2b7685eda5831fbab" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:35.903088 26351 consensus_queue.cc:1035] T 5f745d2dee3d463e9f699080b30f99e5 P 870db067f3704aceba4f9bae8a294d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "748c8722e114457bbb7fdd64d4548e0f" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:35.905253 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.904644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:47607)
0806 16:12:35.905099 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.905166 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
I20260806 16:12:35.905853 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.905564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:59685)
0806 16:12:35.905714 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.905772 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 16:12:35.909221 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.908905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:49425)
0806 16:12:35.909059 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.909138 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 16:12:35.910019 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.909752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.1:32933)
0806 16:12:35.909883 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.909936 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 16:12:35.910430 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.909981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.3:58411)
0806 16:12:35.910298 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.910365 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260806 16:12:35.912055 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.911754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32877 (local address 127.18.90.2:45333)
0806 16:12:35.911905 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.911964 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 16:12:35.919875 25787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:35.920076 25787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:35.924497 25787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5fe45c2c8f4406a8cf84e1adc035c35: Sending DeleteTablet for 3 replicas of tablet 5f745d2dee3d463e9f699080b30f99e5
I20260806 16:12:35.925402 25910 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f745d2dee3d463e9f699080b30f99e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:35 UTC) from {username='oryx', principal='oryx/127.18.90.60@KRBTEST.COM'} at 127.0.0.1:54780
I20260806 16:12:35.925441 26048 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f745d2dee3d463e9f699080b30f99e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:35 UTC) from {username='oryx', principal='oryx/127.18.90.60@KRBTEST.COM'} at 127.0.0.1:43030
I20260806 16:12:35.925835 26184 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f745d2dee3d463e9f699080b30f99e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:35 UTC) from {username='oryx', principal='oryx/127.18.90.60@KRBTEST.COM'} at 127.0.0.1:35046
I20260806 16:12:35.925942 26370 tablet_replica.cc:330] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: stopping tablet replica
I20260806 16:12:35.926082 26370 raft_consensus.cc:2229] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:35.926244 26370 raft_consensus.cc:2258] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:35.926280 26371 tablet_replica.cc:330] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: stopping tablet replica
I20260806 16:12:35.926735 26371 raft_consensus.cc:2229] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:35.926926 26371 raft_consensus.cc:2258] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:35.927440 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25756
I20260806 16:12:35.928992 26371 ts_tablet_manager.cc:1905] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:35.930452 26370 ts_tablet_manager.cc:1905] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:35.931730 26371 ts_tablet_manager.cc:1918] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:35.931831 26371 log.cc:1196] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/5f745d2dee3d463e9f699080b30f99e5
I20260806 16:12:35.932164 26371 ts_tablet_manager.cc:1939] T 5f745d2dee3d463e9f699080b30f99e5 P 748c8722e114457bbb7fdd64d4548e0f: Deleting consensus metadata
I20260806 16:12:35.932994 25779 catalog_manager.cc:4841] TS 748c8722e114457bbb7fdd64d4548e0f (127.18.90.3:39809): tablet 5f745d2dee3d463e9f699080b30f99e5 (table test-table [id=6552fe5b152b42f0837d3a3889c64e10]) successfully deleted
I20260806 16:12:35.933913 26370 ts_tablet_manager.cc:1918] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:35.934010 26370 log.cc:1196] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/5f745d2dee3d463e9f699080b30f99e5
I20260806 16:12:35.934291 26370 ts_tablet_manager.cc:1939] T 5f745d2dee3d463e9f699080b30f99e5 P a5b3b108823940d2b7685eda5831fbab: Deleting consensus metadata
I20260806 16:12:35.935221 25780 catalog_manager.cc:4841] TS a5b3b108823940d2b7685eda5831fbab (127.18.90.2:38549): tablet 5f745d2dee3d463e9f699080b30f99e5 (table test-table [id=6552fe5b152b42f0837d3a3889c64e10]) successfully deleted
I20260806 16:12:35.937075 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25984
I20260806 16:12:35.937417 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:35.937071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45227 (local address 127.0.0.1:54800)
0806 16:12:35.937240 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:35.937308 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 16:12:35.938292 25779 catalog_manager.cc:4584] TS 870db067f3704aceba4f9bae8a294d1c (127.18.90.1:45227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f745d2dee3d463e9f699080b30f99e5: Network error: Client connection negotiation failed: client connection to 127.18.90.1:45227: connect: Connection refused (error 111)
I20260806 16:12:35.944188 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 26121
I20260806 16:12:35.952745 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25471
I20260806 16:12:35.959607 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 25581
2026-08-06T16:12:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5342 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:36 dist-test-slave-0b58 krb5kdc[26374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:36 dist-test-slave-0b58 krb5kdc[26374](info): set up 2 sockets
Aug 06 16:12:36 dist-test-slave-0b58 krb5kdc[26374](info): commencing operation
krb5kdc: starting...
W20260806 16:12:37.018736 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab.
Entry for principal oryx/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab.
I20260806 16:12:37.150873 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45243
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab
--principal=oryx/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:45243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:37.266428 26387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:37.266769 26387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:37.266855 26387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:37.271183 26387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:37.271294 26387 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 16:12:37.271327 26387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:37.271373 26387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:37.271431 26387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:37.271471 26387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:37.277009 26387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:45243
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.62.keytab
--principal=oryx/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:45243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:37.278213 26387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:37.279346 26387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:37.288537 26393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:37.288525 26392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:37.288622 26387 server_base.cc:1034] running on GCE node
W20260806 16:12:37.288525 26395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:37.289465 26387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:37.290603 26387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:37.291776 26387 hybrid_clock.cc:648] HybridClock initialized: now 1786032757291747 us; error 42 us; skew 500 ppm
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:37.294716 26387 init.cc:377] Logged in from keytab as oryx/127.18.90.62@KRBTEST.COM (short username oryx)
I20260806 16:12:37.295691 26387 webserver.cc:460] Webserver started at http://127.18.90.62:39493/ using document root <none> and password file <none>
I20260806 16:12:37.296243 26387 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:37.296382 26387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:37.296576 26387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:37.298738 26387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "a12ff0590be54fc4880f6ecff0b8d83c"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "426a709420d68122ee76d6cc52326ab0"
server_key_iv: "65283c10fc60ec845fd56b9e0b10d77c"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.299291 26387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "a12ff0590be54fc4880f6ecff0b8d83c"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "426a709420d68122ee76d6cc52326ab0"
server_key_iv: "65283c10fc60ec845fd56b9e0b10d77c"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.303157 26387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 16:12:37.305769 26402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:37.307116 26387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:37.307228 26387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "a12ff0590be54fc4880f6ecff0b8d83c"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "426a709420d68122ee76d6cc52326ab0"
server_key_iv: "65283c10fc60ec845fd56b9e0b10d77c"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.307349 26387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:37.324939 26387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:37.328137 26387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:37.328387 26387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:37.337071 26387 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:45243
I20260806 16:12:37.337090 26465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:45243 every 8 connection(s)
I20260806 16:12:37.338192 26387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:37.341149 26466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:37.346901 26466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Bootstrap starting.
I20260806 16:12:37.348495 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 26387
I20260806 16:12:37.348631 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:37.348956 18792 external_mini_cluster.cc:1428] Setting key 68405abe0afcab08c45cfce67818409a
I20260806 16:12:37.349881 26466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:37.350750 26466 log.cc:824] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:37.353221 26466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: No bootstrap required, opened a new log
I20260806 16:12:37.357659 26466 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45243 } }
I20260806 16:12:37.358034 26466 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:37.358109 26466 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12ff0590be54fc4880f6ecff0b8d83c, State: Initialized, Role: FOLLOWER
I20260806 16:12:37.358629 26466 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45243 } }
I20260806 16:12:37.358770 26466 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:37.358865 26466 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:37.358999 26466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:37.360049 26466 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45243 } }
I20260806 16:12:37.360497 26466 leader_election.cc:302] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12ff0590be54fc4880f6ecff0b8d83c; no voters:
I20260806 16:12:37.360913 26466 leader_election.cc:288] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:37.361022 26471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:37.361208 26471 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [term 1 LEADER]: Becoming Leader. State: Replica: a12ff0590be54fc4880f6ecff0b8d83c, State: Running, Role: LEADER
I20260806 16:12:37.361541 26471 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45243 } }
I20260806 16:12:37.362322 26466 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:37.362953 26473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12ff0590be54fc4880f6ecff0b8d83c. Latest consensus state: current_term: 1 leader_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45243 } } }
I20260806 16:12:37.363114 26473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:37.363348 26472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12ff0590be54fc4880f6ecff0b8d83c" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 45243 } } }
I20260806 16:12:37.363469 26472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:37.365089 26475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:37.366230 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.350181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.18.90.62:45243)
0806 16:12:37.350725 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:37.350746 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:37.350788 (+ 42us) server_negotiation.cc:407] Connection header received
0806 16:12:37.351763 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:37.351803 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:37.352147 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:37.352516 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:37.353491 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.354510 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.355848 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.356438 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.356562 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.359613 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:37.359619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:37.359633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:37.359664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:37.361900 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.362425 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.362434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.362442 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.362590 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.362932 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.362936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.362938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.363939 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:37.364113 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:37.364983 (+ 870us) server_negotiation.cc:299] Negotiation successful
0806 16:12:37.365353 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":187,"threads_started":1}
I20260806 16:12:37.368301 26475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:37.374588 26475 catalog_manager.cc:1347] Generated new cluster ID: 1721120e4a0f42108a8cf124637cad41
I20260806 16:12:37.374670 26475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:37.391441 26475 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:37.392815 26475 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:37.402784 26475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Generated new TSK 0
I20260806 16:12:37.403592 26475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab.
Entry for principal oryx/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab.
I20260806 16:12:37.443249 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45243
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:37.582578 26492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:37.582839 26492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:37.582921 26492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:37.587010 26492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:37.587072 26492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:37.587150 26492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:37.587245 26492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:37.592512 26492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:37.593619 26492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:37.594681 26492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:37.604295 26500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:37.604326 26498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:37.604439 26492 server_base.cc:1034] running on GCE node
W20260806 16:12:37.604300 26497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:37.605039 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:37.605705 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:37.606892 26492 hybrid_clock.cc:648] HybridClock initialized: now 1786032757606875 us; error 27 us; skew 500 ppm
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:37.609663 26492 init.cc:377] Logged in from keytab as oryx/127.18.90.1@KRBTEST.COM (short username oryx)
I20260806 16:12:37.610590 26492 webserver.cc:460] Webserver started at http://127.18.90.1:38845/ using document root <none> and password file <none>
I20260806 16:12:37.611213 26492 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:37.611316 26492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:37.611552 26492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:37.613955 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "9f185700f7054edcbb145527ff14776b"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "9ab40c1c7ff333499da98240ed00bb25"
server_key_iv: "ceab1c44d63fb29a068ec781a53f1628"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.614606 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "9f185700f7054edcbb145527ff14776b"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "9ab40c1c7ff333499da98240ed00bb25"
server_key_iv: "ceab1c44d63fb29a068ec781a53f1628"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.618511 26492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:37.621030 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:37.622251 26492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:37.622416 26492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "9f185700f7054edcbb145527ff14776b"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "9ab40c1c7ff333499da98240ed00bb25"
server_key_iv: "ceab1c44d63fb29a068ec781a53f1628"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.622536 26492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:37.643682 26492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:37.646842 26492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:37.647059 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:37.648733 26492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:37.648811 26492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:37.648890 26492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:37.648942 26492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:37.660372 26492 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:32893
I20260806 16:12:37.661521 26492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:37.661798 26623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:32893 every 8 connection(s)
I20260806 16:12:37.662322 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 26492
I20260806 16:12:37.662451 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:37.662740 18792 external_mini_cluster.cc:1428] Setting key b09e263655d91963b783a86ac72a910f
I20260806 16:12:37.663321 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.649537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:41147 (local address 127.18.90.62:45243)
0806 16:12:37.649702 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:37.649707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:37.650579 (+ 872us) server_negotiation.cc:407] Connection header received
0806 16:12:37.651597 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:37.651603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:37.651675 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:37.651811 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:37.653145 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.654089 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.655363 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.655789 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.655928 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.659144 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:37.659148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:37.659150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:37.659192 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:37.660864 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.661462 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.661464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.661466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.661549 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.662014 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.662021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.662026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.662325 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:37.662425 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:37.663015 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 16:12:37.663153 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:37.664424 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.649933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.18.90.1:41147)
0806 16:12:37.650404 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:37.650457 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:37.651340 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:37.652039 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:37.652050 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:37.652478 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:37.652975 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.652988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.654258 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.654266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.655250 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.655260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.655980 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.655985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.656218 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.656913 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:37.656930 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:37.658898 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:37.660984 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.660990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.661005 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.661347 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.661673 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.661676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.661678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.661879 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.662423 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:37.662430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:37.662724 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 16:12:37.662968 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 16:12:37.663320 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":138,"threads_started":1}
I20260806 16:12:37.665931 26624 heartbeater.cc:347] Connected to a master server at 127.18.90.62:45243
I20260806 16:12:37.666095 26624 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:37.666749 26624 heartbeater.cc:511] Master 127.18.90.62:45243 requested a full tablet report, sending...
I20260806 16:12:37.668525 26418 ts_manager.cc:194] Registered new tserver with Master: 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893)
I20260806 16:12:37.670199 26418 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:41147
WARNING: no policy specified for oryx/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:37.678398 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.668630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:53893 (local address 127.18.90.62:45243)
0806 16:12:37.668764 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:37.668768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:37.671167 (+ 2399us) server_negotiation.cc:407] Connection header received
0806 16:12:37.671320 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:37.671323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:37.671374 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:37.671464 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:37.672068 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.672833 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.673656 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.673890 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.674005 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.675210 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:37.675214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:37.675221 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:37.675250 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:37.676700 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.677173 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.677176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.677177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.677260 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.677509 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.677511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.677512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.677696 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:37.677776 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:37.678153 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 16:12:37.678271 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:12:37.678462 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.668575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.18.90.1:53893)
0806 16:12:37.671057 (+ 2482us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:37.671075 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:37.671206 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:37.671510 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:37.671516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:37.671778 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:37.671928 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.671944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.672965 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.672969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.673528 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.673535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.674016 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.674019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.674120 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.674707 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:37.674718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:37.675050 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:37.676853 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.676856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.676857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.677068 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.677347 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.677349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.677351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.677430 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.677843 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:37.677849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:37.677959 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 16:12:37.678092 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 16:12:37.678251 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab.
Entry for principal oryx/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab.
I20260806 16:12:37.695166 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45243
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:37.809299 26633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:37.809571 26633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:37.809614 26633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:37.813884 26633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:37.813948 26633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:37.813985 26633 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:37.814065 26633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:37.819270 26633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:37.820495 26633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:37.821545 26633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:37.831772 26639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:37.831799 26638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:37.831769 26641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:37.832183 26633 server_base.cc:1034] running on GCE node
I20260806 16:12:37.832650 26633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:37.833307 26633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:37.834470 26633 hybrid_clock.cc:648] HybridClock initialized: now 1786032757834454 us; error 29 us; skew 500 ppm
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:37.837383 26633 init.cc:377] Logged in from keytab as oryx/127.18.90.2@KRBTEST.COM (short username oryx)
I20260806 16:12:37.838266 26633 webserver.cc:460] Webserver started at http://127.18.90.2:34067/ using document root <none> and password file <none>
I20260806 16:12:37.838868 26633 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:37.838963 26633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:37.839205 26633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:37.841450 26633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "50dca885423645b88348e9669a09968a"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "43cc1cf1e109d8c8db26889c6119581b"
server_key_iv: "e262cfefe62d3aea24fefbdcf7226063"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.842072 26633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "50dca885423645b88348e9669a09968a"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "43cc1cf1e109d8c8db26889c6119581b"
server_key_iv: "e262cfefe62d3aea24fefbdcf7226063"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.845955 26633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:37.848537 26648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:37.849772 26633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:37.849926 26633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "50dca885423645b88348e9669a09968a"
format_stamp: "Formatted at 2026-08-06 16:12:37 on dist-test-slave-0b58"
server_key: "43cc1cf1e109d8c8db26889c6119581b"
server_key_iv: "e262cfefe62d3aea24fefbdcf7226063"
server_key_version: "encryptionkey@0"
I20260806 16:12:37.850047 26633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:37.880208 26633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:37.884099 26633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:37.884364 26633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:37.886044 26633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:37.886121 26633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:37.886211 26633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:37.886265 26633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:37 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:37.897809 26633 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:41305
I20260806 16:12:37.897883 26766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:41305 every 8 connection(s)
I20260806 16:12:37.898909 26633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:37.900862 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.886916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:59221 (local address 127.18.90.62:45243)
0806 16:12:37.887068 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:37.887073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:37.888023 (+ 950us) server_negotiation.cc:407] Connection header received
0806 16:12:37.889191 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:37.889194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:37.889257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:37.889375 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:37.891091 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.891635 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.892932 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.893273 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.893407 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.896664 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:37.896668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:37.896670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:37.896697 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:37.898404 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.899049 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.899051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.899053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.899132 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.899558 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.899564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.899568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.899834 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:37.899957 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:37.900550 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 16:12:37.900684 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:37.901674 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.887380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.18.90.2:59221)
0806 16:12:37.887845 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:37.887897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:37.888918 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:37.889594 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:37.889607 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:37.890116 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:37.890897 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.890926 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.891779 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.891784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.892791 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.892801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.893461 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.893465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.893714 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.894395 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:37.894411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:37.896465 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:37.898528 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.898535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.898551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.898902 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.899232 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.899238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.899243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.899435 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.899954 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:37.899962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:37.900225 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 16:12:37.900486 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 16:12:37.900830 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260806 16:12:37.903126 26767 heartbeater.cc:347] Connected to a master server at 127.18.90.62:45243
I20260806 16:12:37.903301 26767 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:37.903461 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 26633
I20260806 16:12:37.903585 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:37.903908 18792 external_mini_cluster.cc:1428] Setting key 69e636dbcb23f2e2f10ca2b64b337231
I20260806 16:12:37.903918 26767 heartbeater.cc:511] Master 127.18.90.62:45243 requested a full tablet report, sending...
I20260806 16:12:37.905128 26418 ts_manager.cc:194] Registered new tserver with Master: 50dca885423645b88348e9669a09968a (127.18.90.2:41305)
I20260806 16:12:37.905920 26418 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:59221
I20260806 16:12:37.913950 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.905525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:50629 (local address 127.18.90.62:45243)
0806 16:12:37.905713 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:37.905717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:37.905811 (+ 94us) server_negotiation.cc:407] Connection header received
0806 16:12:37.905933 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:37.905936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:37.905973 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:37.906063 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:37.906692 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.907429 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.908734 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.909116 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.909234 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.910551 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:37.910553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:37.910555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:37.910577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:37.912044 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.912634 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.912638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.912639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.912745 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:37.913080 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:37.913082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:37.913083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:37.913301 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:37.913383 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:37.913718 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 16:12:37.913829 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 16:12:37.914405 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:37.905418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.18.90.2:50629)
0806 16:12:37.905725 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:37.905740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:37.905838 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:37.906127 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:37.906130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:37.906388 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:37.906515 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.906524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.907581 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.907587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.908599 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:37.908615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:37.909242 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:37.909244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:37.909337 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:37.909916 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:37.909936 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:37.910426 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:37.912148 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.912151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.912153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.912504 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.912860 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:37.912865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:37.912870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:37.912978 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:37.913407 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:37.913413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:37.913536 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 16:12:37.913672 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 16:12:37.913846 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab.
Entry for principal oryx/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab.
I20260806 16:12:37.935559 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:45243
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:38.050117 26776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:38.050436 26776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:38.050523 26776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:38.054687 26776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:38.054769 26776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:38.054839 26776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:12:38.054917 26776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:38.060295 26776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.90.62:45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:38.061477 26776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:38.062651 26776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:38.072564 26783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:38.072556 26782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:38.072556 26785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:38.072868 26776 server_base.cc:1034] running on GCE node
I20260806 16:12:38.073282 26776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:38.073896 26776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:38.075089 26776 hybrid_clock.cc:648] HybridClock initialized: now 1786032758075075 us; error 39 us; skew 500 ppm
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032758, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:38.077852 26776 init.cc:377] Logged in from keytab as oryx/127.18.90.3@KRBTEST.COM (short username oryx)
I20260806 16:12:38.078706 26776 webserver.cc:460] Webserver started at http://127.18.90.3:38075/ using document root <none> and password file <none>
I20260806 16:12:38.079279 26776 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:38.079370 26776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:38.079595 26776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:38.081845 26776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "36b67ca47f594bf1bb6b87e72954355b"
format_stamp: "Formatted at 2026-08-06 16:12:38 on dist-test-slave-0b58"
server_key: "2fd214da859a1bf7824d36f8a7858a29"
server_key_iv: "0f97729bef96a99d45eeba0a5022cd19"
server_key_version: "encryptionkey@0"
I20260806 16:12:38.082401 26776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "36b67ca47f594bf1bb6b87e72954355b"
format_stamp: "Formatted at 2026-08-06 16:12:38 on dist-test-slave-0b58"
server_key: "2fd214da859a1bf7824d36f8a7858a29"
server_key_iv: "0f97729bef96a99d45eeba0a5022cd19"
server_key_version: "encryptionkey@0"
I20260806 16:12:38.086199 26776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:38.088578 26792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:38.089782 26776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:12:38.089910 26776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "36b67ca47f594bf1bb6b87e72954355b"
format_stamp: "Formatted at 2026-08-06 16:12:38 on dist-test-slave-0b58"
server_key: "2fd214da859a1bf7824d36f8a7858a29"
server_key_iv: "0f97729bef96a99d45eeba0a5022cd19"
server_key_version: "encryptionkey@0"
I20260806 16:12:38.090019 26776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:38.104468 26776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:38.107370 26776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:38.107525 26776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:38.109018 26776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:38.109094 26776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:38.109179 26776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:38.109225 26776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032758, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:38.120545 26776 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:44145
I20260806 16:12:38.121549 26776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:38.121743 26907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:44145 every 8 connection(s)
I20260806 16:12:38.122390 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 26776
I20260806 16:12:38.122530 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:38.122895 18792 external_mini_cluster.cc:1428] Setting key 05f83ef0afb031dda8671cd28dafa003
I20260806 16:12:38.124071 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.109705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:52063 (local address 127.18.90.62:45243)
0806 16:12:38.109869 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.109874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.110737 (+ 863us) server_negotiation.cc:407] Connection header received
0806 16:12:38.111745 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.111749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.111818 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.111958 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:38.113393 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.114320 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.116296 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.116656 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.116767 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.119820 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:38.119823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:38.119825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:38.119859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:38.121578 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.122236 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.122241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.122243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.122354 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.122792 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.122795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.122796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.123043 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:38.123134 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.123756 (+ 622us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.123901 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:12:38.125047 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.110099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.18.90.3:52063)
0806 16:12:38.110569 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.110619 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.111525 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.112139 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.112150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.112652 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:38.113236 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.113252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.114501 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.114507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.116121 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.116135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.116851 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.116856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.117094 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.117705 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:38.117727 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:38.119628 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:38.121724 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:38.121732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:38.121748 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:38.122109 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:38.122472 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:38.122476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:38.122480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:38.122662 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:38.123161 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:38.123167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:38.123470 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.123701 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.124023 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
I20260806 16:12:38.126216 26908 heartbeater.cc:347] Connected to a master server at 127.18.90.62:45243
I20260806 16:12:38.126381 26908 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:38.126945 26908 heartbeater.cc:511] Master 127.18.90.62:45243 requested a full tablet report, sending...
I20260806 16:12:38.128055 26418 ts_manager.cc:194] Registered new tserver with Master: 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.128772 26418 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:52063
I20260806 16:12:38.135335 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.128510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.18.90.3:42779)
0806 16:12:38.128934 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.128955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.129072 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.129443 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.129446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.129668 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:38.129783 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.129791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.130523 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.130526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.131208 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.131217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.131681 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.131684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.131747 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.132179 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:38.132190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:38.132564 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:38.134049 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:38.134052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:38.134053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:38.134244 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:38.134498 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:38.134500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:38.134501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:38.134575 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:38.134894 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:38.134896 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:38.135014 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.135091 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.135195 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":241,"threads_started":1}
I20260806 16:12:38.135344 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.128616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:42779 (local address 127.18.90.62:45243)
0806 16:12:38.128725 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.128729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.129029 (+ 300us) server_negotiation.cc:407] Connection header received
0806 16:12:38.129187 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.129190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.129240 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.129286 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:38.129901 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.130387 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.131343 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.131580 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.131678 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.132683 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:38.132685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:38.132687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:38.132709 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:38.133955 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.134346 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.134349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.134350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.134420 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.134649 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.134651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.134652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.134817 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:38.134889 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.135147 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.135235 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 16:12:38.137264 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:12:38.162142 26469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:38.155482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.18.90.62:45243)
0806 16:12:38.155617 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.155621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.155634 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:38.155680 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.155683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.155723 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.155811 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:38.156561 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.157085 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.157964 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.158225 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.158326 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.161934 (+ 3608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46806: BlockingRecv error: recv got EOF from 127.0.0.1:46806 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:38.172642 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.163924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.18.90.62:45243)
0806 16:12:38.164069 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.164073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.164245 (+ 172us) server_negotiation.cc:407] Connection header received
0806 16:12:38.164408 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.164410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.164451 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.164557 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:38.165253 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.165748 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.166637 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.166850 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.166924 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.169391 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:38.169394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:38.169396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:38.169423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:38.170675 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.171097 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.171100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.171102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.171195 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.171684 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.171687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.171688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.171876 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:38.171988 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.172226 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.172428 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:12:38.175827 26418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:38.178946 26418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:38.193748 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.187686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:32893 (local address 127.0.0.1:53112)
0806 16:12:38.188465 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.188481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.188670 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.189404 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.189410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.189448 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.189769 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.189776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.190752 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.190755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.192081 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.192092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.193184 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.193186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.193289 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.193316 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.193444 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.193574 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":96,"spinlock_wait_cycles":6144,"thread_start_us":100,"threads_started":1}
I20260806 16:12:38.193914 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.188717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.18.90.3:44145)
0806 16:12:38.189198 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.189210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.189231 (+ 21us) server_negotiation.cc:407] Connection header received
0806 16:12:38.189306 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.189310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.189515 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.189669 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.189925 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.191106 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.192549 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.193291 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.193438 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.193568 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.193626 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.193725 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":126,"threads_started":1}
I20260806 16:12:38.194054 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.187469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:41305 (local address 127.0.0.1:34046)
0806 16:12:38.188409 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.188467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.188614 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.189404 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.189410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.189446 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.189758 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.189767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.191256 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.191259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.192336 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.192348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.193636 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.193639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.193735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.193760 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.193831 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.193897 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":108,"spinlock_wait_cycles":5248,"thread_start_us":121,"threads_started":1}
I20260806 16:12:38.194392 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.187866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.18.90.1:32893)
0806 16:12:38.188176 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.188183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.188588 (+ 405us) server_negotiation.cc:407] Connection header received
0806 16:12:38.188844 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.188849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.189074 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.189223 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.189872 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.190623 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.192226 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.192928 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.192993 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.193104 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.193843 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.194239 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
I20260806 16:12:38.193758 26934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.188276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44145 (local address 127.0.0.1:33618)
0806 16:12:38.188528 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.188541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.188613 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.189694 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.189697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.189714 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.189814 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.189823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.191237 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.191240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.192444 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.192456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.193424 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.193430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.193511 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.193530 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.193586 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.193639 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":31,"thread_start_us":93,"threads_started":1}
I20260806 16:12:38.194590 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.187618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.18.90.2:41305)
0806 16:12:38.188326 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.188342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.188754 (+ 412us) server_negotiation.cc:407] Connection header received
0806 16:12:38.188842 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.188847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.189074 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.189222 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.189903 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.191106 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.192474 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.193291 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.193401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.193569 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.194337 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.194453 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":140,"threads_started":1}
I20260806 16:12:38.197224 26701 tablet_service.cc:1467] Processing CreateTablet for tablet edd91f238a8445c4bb6a034de69e1449 (DEFAULT_TABLE table=test-table [id=ad2f718c2acc42dc8f44ce1c31228a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:38.197291 26842 tablet_service.cc:1467] Processing CreateTablet for tablet edd91f238a8445c4bb6a034de69e1449 (DEFAULT_TABLE table=test-table [id=ad2f718c2acc42dc8f44ce1c31228a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:38.197273 26558 tablet_service.cc:1467] Processing CreateTablet for tablet edd91f238a8445c4bb6a034de69e1449 (DEFAULT_TABLE table=test-table [id=ad2f718c2acc42dc8f44ce1c31228a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:38.198225 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd91f238a8445c4bb6a034de69e1449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.198282 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd91f238a8445c4bb6a034de69e1449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.198285 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd91f238a8445c4bb6a034de69e1449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.205466 26936 tablet_bootstrap.cc:492] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Bootstrap starting.
I20260806 16:12:38.206382 26938 tablet_bootstrap.cc:492] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Bootstrap starting.
I20260806 16:12:38.206874 26937 tablet_bootstrap.cc:492] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Bootstrap starting.
I20260806 16:12:38.207826 26936 tablet_bootstrap.cc:654] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.208561 26938 tablet_bootstrap.cc:654] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.208668 26936 log.cc:824] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:38.208914 26937 tablet_bootstrap.cc:654] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.209425 26938 log.cc:824] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:38.209683 26937 log.cc:824] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:38.211107 26936 tablet_bootstrap.cc:492] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: No bootstrap required, opened a new log
I20260806 16:12:38.211367 26936 ts_tablet_manager.cc:1397] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:12:38.211442 26938 tablet_bootstrap.cc:492] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: No bootstrap required, opened a new log
I20260806 16:12:38.211433 26937 tablet_bootstrap.cc:492] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: No bootstrap required, opened a new log
I20260806 16:12:38.211623 26938 ts_tablet_manager.cc:1397] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:12:38.211623 26937 ts_tablet_manager.cc:1397] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 16:12:38.214442 26936 raft_consensus.cc:348] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.214689 26936 raft_consensus.cc:374] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.214773 26936 raft_consensus.cc:729] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dca885423645b88348e9669a09968a, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.215240 26936 consensus_queue.cc:260] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.215960 26767 heartbeater.cc:503] Master 127.18.90.62:45243 was elected leader, sending a full tablet report...
I20260806 16:12:38.216023 26937 raft_consensus.cc:348] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.216300 26937 raft_consensus.cc:374] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.216365 26937 raft_consensus.cc:729] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.216357 26936 ts_tablet_manager.cc:1428] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260806 16:12:38.216856 26937 consensus_queue.cc:260] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.215667 26938 raft_consensus.cc:348] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.217155 26938 raft_consensus.cc:374] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.217211 26938 raft_consensus.cc:729] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b67ca47f594bf1bb6b87e72954355b, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.217623 26938 consensus_queue.cc:260] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.218076 26937 ts_tablet_manager.cc:1428] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 16:12:38.218334 26908 heartbeater.cc:503] Master 127.18.90.62:45243 was elected leader, sending a full tablet report...
I20260806 16:12:38.218557 26938 ts_tablet_manager.cc:1428] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 16:12:38.219357 26624 heartbeater.cc:503] Master 127.18.90.62:45243 was elected leader, sending a full tablet report...
I20260806 16:12:38.294342 26943 raft_consensus.cc:482] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:38.294585 26943 raft_consensus.cc:504] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.295894 26943 leader_election.cc:288] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50dca885423645b88348e9669a09968a (127.18.90.2:41305), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.299979 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.296430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:57165 (local address 127.18.90.2:41305)
0806 16:12:38.296559 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.296563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.296737 (+ 174us) server_negotiation.cc:407] Connection header received
0806 16:12:38.296889 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.296892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.296945 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.297026 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.297337 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.297869 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.299010 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.299410 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.299528 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.299590 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.299776 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.299846 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:12:38.300007 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.296321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:56333 (local address 127.18.90.3:44145)
0806 16:12:38.296496 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.296501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.296821 (+ 320us) server_negotiation.cc:407] Connection header received
0806 16:12:38.296943 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.296946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.296996 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.297083 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.297391 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.297892 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.299018 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.299412 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.299529 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.299594 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.299808 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.299877 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 16:12:38.300019 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.296161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:41305 (local address 127.18.90.1:57165)
0806 16:12:38.296636 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.296648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.296786 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.297049 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.297052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.297068 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.297210 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.297216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.297982 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.297988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.298874 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.298883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.299536 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.299539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.299636 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.299666 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.299756 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.299827 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":21,"thread_start_us":152,"threads_started":1}
I20260806 16:12:38.300033 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.296161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44145 (local address 127.18.90.1:56333)
0806 16:12:38.296681 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.296693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.296845 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.297096 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.297099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.297114 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.297270 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.297276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.298011 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.298015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.298884 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.298892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.299545 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.299549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.299669 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.299696 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.299790 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.299859 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":215,"thread_start_us":123,"threads_started":1}
I20260806 16:12:38.300747 26721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd91f238a8445c4bb6a034de69e1449" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50dca885423645b88348e9669a09968a" is_pre_election: true
I20260806 16:12:38.301005 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd91f238a8445c4bb6a034de69e1449" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b" is_pre_election: true
I20260806 16:12:38.301074 26721 raft_consensus.cc:2454] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 0.
I20260806 16:12:38.301280 26862 raft_consensus.cc:2454] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 0.
I20260806 16:12:38.301615 26509 leader_election.cc:302] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50dca885423645b88348e9669a09968a, 9f185700f7054edcbb145527ff14776b; no voters:
I20260806 16:12:38.301908 26943 raft_consensus.cc:2781] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:38.301988 26943 raft_consensus.cc:482] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:38.302019 26943 raft_consensus.cc:3037] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.303217 26943 raft_consensus.cc:504] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.303592 26943 leader_election.cc:288] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 election: Requested vote from peers 50dca885423645b88348e9669a09968a (127.18.90.2:41305), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.303969 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd91f238a8445c4bb6a034de69e1449" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b"
I20260806 16:12:38.303970 26721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd91f238a8445c4bb6a034de69e1449" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50dca885423645b88348e9669a09968a"
I20260806 16:12:38.304097 26721 raft_consensus.cc:3037] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.304097 26862 raft_consensus.cc:3037] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.305467 26862 raft_consensus.cc:2454] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 1.
I20260806 16:12:38.305467 26721 raft_consensus.cc:2454] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 1.
I20260806 16:12:38.305802 26509 leader_election.cc:302] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 50dca885423645b88348e9669a09968a, 9f185700f7054edcbb145527ff14776b; no voters:
I20260806 16:12:38.305948 26943 raft_consensus.cc:2781] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:38.306169 26943 raft_consensus.cc:686] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 1 LEADER]: Becoming Leader. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Running, Role: LEADER
I20260806 16:12:38.306555 26943 consensus_queue.cc:237] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.310060 26419 catalog_manager.cc:5495] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f185700f7054edcbb145527ff14776b (127.18.90.1). New cstate: current_term: 1 leader_uuid: "9f185700f7054edcbb145527ff14776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
I20260806 16:12:38.347905 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.344140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.18.90.1:32893)
0806 16:12:38.344317 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.344322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.344339 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:12:38.344507 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.344513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.344592 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.344730 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:38.344989 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.345522 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.346427 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.346656 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.346769 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.347064 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:38.347497 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:38.347629 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.347692 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.347755 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:12:38.354146 26721 raft_consensus.cc:1261] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Refusing update from remote peer 9f185700f7054edcbb145527ff14776b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:38.354496 26862 raft_consensus.cc:1261] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Refusing update from remote peer 9f185700f7054edcbb145527ff14776b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:38.354877 26943 consensus_queue.cc:1035] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:38.355126 26947 consensus_queue.cc:1035] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:38.363163 26952 mvcc.cc:205] Tried to move back new op lower bound from 7315590178210979840 to 7315590178026655744. Current Snapshot: MvccSnapshot[applied={T|T < 7315590178210979840}]
I20260806 16:12:38.371558 26418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:38.371770 26418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:38.374400 26418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Sending DeleteTablet for 3 replicas of tablet edd91f238a8445c4bb6a034de69e1449
I20260806 16:12:38.375366 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet edd91f238a8445c4bb6a034de69e1449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:38 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260806 16:12:38.375490 26701 tablet_service.cc:1514] Processing DeleteTablet for tablet edd91f238a8445c4bb6a034de69e1449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:38 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34046
I20260806 16:12:38.375885 26960 tablet_replica.cc:330] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: stopping tablet replica
I20260806 16:12:38.375991 26961 tablet_replica.cc:330] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: stopping tablet replica
I20260806 16:12:38.376009 26558 tablet_service.cc:1514] Processing DeleteTablet for tablet edd91f238a8445c4bb6a034de69e1449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:38 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:53112
I20260806 16:12:38.376333 26960 raft_consensus.cc:2229] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:38.376333 26961 raft_consensus.cc:2229] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:38.376654 26961 raft_consensus.cc:2258] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:38.376654 26960 raft_consensus.cc:2258] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:38.377017 26962 tablet_replica.cc:330] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: stopping tablet replica
I20260806 16:12:38.377149 26418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 16:12:38.377315 26962 raft_consensus.cc:2229] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 1 LEADER]: Raft consensus shutting down.
W20260806 16:12:38.377581 26418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:38.377703 26962 raft_consensus.cc:2258] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:38.379153 26962 ts_tablet_manager.cc:1905] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:38.380038 26960 ts_tablet_manager.cc:1905] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:38.380797 26961 ts_tablet_manager.cc:1905] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:38.382064 26962 ts_tablet_manager.cc:1918] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:38.382166 26962 log.cc:1196] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/edd91f238a8445c4bb6a034de69e1449
I20260806 16:12:38.382473 26962 ts_tablet_manager.cc:1939] T edd91f238a8445c4bb6a034de69e1449 P 9f185700f7054edcbb145527ff14776b: Deleting consensus metadata
I20260806 16:12:38.383709 26842 tablet_service.cc:1467] Processing CreateTablet for tablet c9139acb98d64085afebe89a40ea90dd (DEFAULT_TABLE table=test-table [id=b59fc88764ef48be88d363fcd0570257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:38.383785 26558 tablet_service.cc:1467] Processing CreateTablet for tablet c9139acb98d64085afebe89a40ea90dd (DEFAULT_TABLE table=test-table [id=b59fc88764ef48be88d363fcd0570257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:38.383881 26406 catalog_manager.cc:4841] TS 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893): tablet edd91f238a8445c4bb6a034de69e1449 (table test-table [id=ad2f718c2acc42dc8f44ce1c31228a1a]) successfully deleted
I20260806 16:12:38.383989 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9139acb98d64085afebe89a40ea90dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.384027 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9139acb98d64085afebe89a40ea90dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.386106 26938 tablet_bootstrap.cc:492] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Bootstrap starting.
I20260806 16:12:38.386433 26701 tablet_service.cc:1467] Processing CreateTablet for tablet c9139acb98d64085afebe89a40ea90dd (DEFAULT_TABLE table=test-table [id=b59fc88764ef48be88d363fcd0570257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:38.386684 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9139acb98d64085afebe89a40ea90dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.387064 26938 tablet_bootstrap.cc:654] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.388334 26938 tablet_bootstrap.cc:492] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: No bootstrap required, opened a new log
I20260806 16:12:38.388404 26937 tablet_bootstrap.cc:492] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Bootstrap starting.
I20260806 16:12:38.388422 26938 ts_tablet_manager.cc:1397] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:38.388922 26938 raft_consensus.cc:348] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.389043 26938 raft_consensus.cc:374] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.389070 26938 raft_consensus.cc:729] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b67ca47f594bf1bb6b87e72954355b, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.389201 26938 consensus_queue.cc:260] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.389365 26937 tablet_bootstrap.cc:654] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.389465 26938 ts_tablet_manager.cc:1428] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:38.391012 26936 tablet_bootstrap.cc:492] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Bootstrap starting.
I20260806 16:12:38.391696 26937 tablet_bootstrap.cc:492] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: No bootstrap required, opened a new log
I20260806 16:12:38.391800 26937 ts_tablet_manager.cc:1397] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 16:12:38.392076 26936 tablet_bootstrap.cc:654] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.392292 26937 raft_consensus.cc:348] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.392414 26937 raft_consensus.cc:374] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.392452 26937 raft_consensus.cc:729] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.392580 26937 consensus_queue.cc:260] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.392858 26937 ts_tablet_manager.cc:1428] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:38.393463 26960 ts_tablet_manager.cc:1918] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:38.394065 26961 ts_tablet_manager.cc:1918] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:38.394590 26960 log.cc:1196] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/edd91f238a8445c4bb6a034de69e1449
I20260806 16:12:38.394734 26961 log.cc:1196] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/edd91f238a8445c4bb6a034de69e1449
I20260806 16:12:38.395350 26961 ts_tablet_manager.cc:1939] T edd91f238a8445c4bb6a034de69e1449 P 50dca885423645b88348e9669a09968a: Deleting consensus metadata
I20260806 16:12:38.395653 26960 ts_tablet_manager.cc:1939] T edd91f238a8445c4bb6a034de69e1449 P 36b67ca47f594bf1bb6b87e72954355b: Deleting consensus metadata
I20260806 16:12:38.395794 26936 tablet_bootstrap.cc:492] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: No bootstrap required, opened a new log
I20260806 16:12:38.396926 26936 ts_tablet_manager.cc:1397] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 16:12:38.396919 26406 catalog_manager.cc:4841] TS 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145): tablet edd91f238a8445c4bb6a034de69e1449 (table test-table [id=ad2f718c2acc42dc8f44ce1c31228a1a]) successfully deleted
I20260806 16:12:38.397334 26936 raft_consensus.cc:348] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.397454 26936 raft_consensus.cc:374] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.397480 26936 raft_consensus.cc:729] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dca885423645b88348e9669a09968a, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.397620 26936 consensus_queue.cc:260] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.397900 26936 ts_tablet_manager.cc:1428] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:38.399405 26404 catalog_manager.cc:4841] TS 50dca885423645b88348e9669a09968a (127.18.90.2:41305): tablet edd91f238a8445c4bb6a034de69e1449 (table test-table [id=ad2f718c2acc42dc8f44ce1c31228a1a]) successfully deleted
W20260806 16:12:38.400077 26768 tablet.cc:2369] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:12:38.413179 26625 tablet.cc:2369] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:38.616600 26955 raft_consensus.cc:482] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:38.616762 26955 raft_consensus.cc:504] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.617162 26955 leader_election.cc:288] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50dca885423645b88348e9669a09968a (127.18.90.2:41305), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.617774 26721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9139acb98d64085afebe89a40ea90dd" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50dca885423645b88348e9669a09968a" is_pre_election: true
I20260806 16:12:38.617779 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9139acb98d64085afebe89a40ea90dd" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b" is_pre_election: true
I20260806 16:12:38.617944 26862 raft_consensus.cc:2454] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 0.
I20260806 16:12:38.617944 26721 raft_consensus.cc:2454] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 0.
I20260806 16:12:38.618314 26511 leader_election.cc:302] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 9f185700f7054edcbb145527ff14776b; no voters:
I20260806 16:12:38.618521 26955 raft_consensus.cc:2781] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:38.618582 26955 raft_consensus.cc:482] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:38.618609 26955 raft_consensus.cc:3037] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.619684 26955 raft_consensus.cc:504] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.620023 26955 leader_election.cc:288] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 election: Requested vote from peers 50dca885423645b88348e9669a09968a (127.18.90.2:41305), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.620436 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9139acb98d64085afebe89a40ea90dd" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b"
I20260806 16:12:38.620440 26721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9139acb98d64085afebe89a40ea90dd" candidate_uuid: "9f185700f7054edcbb145527ff14776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50dca885423645b88348e9669a09968a"
I20260806 16:12:38.620558 26862 raft_consensus.cc:3037] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.620601 26721 raft_consensus.cc:3037] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.621526 26721 raft_consensus.cc:2454] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 1.
I20260806 16:12:38.621529 26862 raft_consensus.cc:2454] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f185700f7054edcbb145527ff14776b in term 1.
I20260806 16:12:38.621847 26509 leader_election.cc:302] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50dca885423645b88348e9669a09968a, 9f185700f7054edcbb145527ff14776b; no voters:
I20260806 16:12:38.622061 26955 raft_consensus.cc:2781] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:38.622159 26955 raft_consensus.cc:686] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 1 LEADER]: Becoming Leader. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Running, Role: LEADER
I20260806 16:12:38.622318 26955 consensus_queue.cc:237] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
W20260806 16:12:38.623451 26909 tablet.cc:2369] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:12:38.624032 26416 catalog_manager.cc:5495] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f185700f7054edcbb145527ff14776b (127.18.90.1). New cstate: current_term: 1 leader_uuid: "9f185700f7054edcbb145527ff14776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
Aug 06 16:12:38 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032757, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.62@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:38.710830 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.702066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:45243 (local address 127.0.0.1:46822)
0806 16:12:38.702462 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.702483 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.702627 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.702901 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.702904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.703412 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:38.703519 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.703527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.704382 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.704389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.704961 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.704969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.705442 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.705445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.705535 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.706161 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:38.706173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:38.707678 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:38.709339 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:38.709348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:38.709353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:38.709596 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:38.709861 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:38.709863 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:38.709864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:38.709944 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:38.710289 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:38.710295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:38.710463 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.710562 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.710694 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":117,"threads_started":1}
I20260806 16:12:38.710906 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.702102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.18.90.62:45243)
0806 16:12:38.702425 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.702437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.702578 (+ 141us) server_negotiation.cc:407] Connection header received
0806 16:12:38.702733 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.702736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.702792 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.702885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:38.703643 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.704249 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.705080 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.705317 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.705424 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.707831 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:38.707834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:38.707836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:38.707858 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:38.709217 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.709689 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.709692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.709693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.709779 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:38.710021 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:38.710023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:38.710024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:38.710194 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:38.710271 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.710616 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.710753 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260806 16:12:38.714570 26416 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:46822:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 16:12:38.715312 26416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:38.720023 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 81969afbb56a4502a8f202557f0db2c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37dfac86f3114e5abaf4954fddb10bad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:38.720033 26701 tablet_service.cc:1467] Processing CreateTablet for tablet 81969afbb56a4502a8f202557f0db2c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37dfac86f3114e5abaf4954fddb10bad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:38.720324 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 81969afbb56a4502a8f202557f0db2c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37dfac86f3114e5abaf4954fddb10bad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:12:38.720494 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81969afbb56a4502a8f202557f0db2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.720497 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81969afbb56a4502a8f202557f0db2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.720664 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81969afbb56a4502a8f202557f0db2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:38.722842 26937 tablet_bootstrap.cc:492] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: Bootstrap starting.
I20260806 16:12:38.723490 26936 tablet_bootstrap.cc:492] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: Bootstrap starting.
I20260806 16:12:38.723763 26937 tablet_bootstrap.cc:654] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.723984 26938 tablet_bootstrap.cc:492] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: Bootstrap starting.
I20260806 16:12:38.724514 26936 tablet_bootstrap.cc:654] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.724987 26938 tablet_bootstrap.cc:654] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:38.725119 26937 tablet_bootstrap.cc:492] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: No bootstrap required, opened a new log
I20260806 16:12:38.725245 26937 ts_tablet_manager.cc:1397] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:38.725656 26937 raft_consensus.cc:348] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.725804 26937 raft_consensus.cc:374] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.725848 26937 raft_consensus.cc:729] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.725988 26937 consensus_queue.cc:260] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.726312 26937 ts_tablet_manager.cc:1428] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:12:38.726327 26938 tablet_bootstrap.cc:492] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: No bootstrap required, opened a new log
I20260806 16:12:38.726261 26955 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:38.726401 26938 ts_tablet_manager.cc:1397] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:38.726408 26955 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:38.726883 26938 raft_consensus.cc:348] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.727010 26938 raft_consensus.cc:374] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.727049 26938 raft_consensus.cc:729] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b67ca47f594bf1bb6b87e72954355b, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.727180 26938 consensus_queue.cc:260] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.727473 26938 ts_tablet_manager.cc:1428] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:38.727438 26944 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:38.727591 26944 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:38.728813 26936 tablet_bootstrap.cc:492] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: No bootstrap required, opened a new log
I20260806 16:12:38.728924 26936 ts_tablet_manager.cc:1397] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 16:12:38.729393 26936 raft_consensus.cc:348] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.729524 26936 raft_consensus.cc:374] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:38.729566 26936 raft_consensus.cc:729] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dca885423645b88348e9669a09968a, State: Initialized, Role: FOLLOWER
I20260806 16:12:38.729712 26936 consensus_queue.cc:260] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.729990 26942 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:38.730165 26942 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:38.730435 26936 ts_tablet_manager.cc:1428] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:38.952510 26942 raft_consensus.cc:482] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:38.952689 26942 raft_consensus.cc:504] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.953814 26942 leader_election.cc:288] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.958158 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.954258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:42081 (local address 127.18.90.1:32893)
0806 16:12:38.954578 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.954583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.954602 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:12:38.954763 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.954768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.954836 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.954964 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.955303 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.955914 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.957134 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.957578 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.957659 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.957714 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.957942 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.958017 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260806 16:12:38.958153 26978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.954188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:32893 (local address 127.18.90.2:42081)
0806 16:12:38.954472 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.954511 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.954640 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.955009 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.955013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.955042 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.955173 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.955183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.956026 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.956030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.957003 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.957013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.957713 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.957719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.957820 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.957843 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.957920 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.957997 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":17,"thread_start_us":83,"threads_started":1}
I20260806 16:12:38.958436 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.954753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:59849 (local address 127.18.90.3:44145)
0806 16:12:38.955009 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:38.955014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:38.955028 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:38.955079 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:38.955082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:38.955127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:38.955261 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:38.955539 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.956096 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.957240 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.957679 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.957783 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.957847 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:38.958205 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 16:12:38.958298 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260806 16:12:38.958689 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:38.954610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44145 (local address 127.18.90.2:59849)
0806 16:12:38.954847 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:38.954860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:38.954933 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:38.955304 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:38.955306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:38.955323 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:38.955426 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.955432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.956233 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.956236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.957113 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:38.957125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:38.957845 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:38.957849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:38.957941 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:38.957965 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:12:38.958038 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 16:12:38.958114 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":17,"thread_start_us":59,"threads_started":1}
I20260806 16:12:38.958906 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81969afbb56a4502a8f202557f0db2c2" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f185700f7054edcbb145527ff14776b" is_pre_election: true
I20260806 16:12:38.959093 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81969afbb56a4502a8f202557f0db2c2" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b" is_pre_election: true
I20260806 16:12:38.959198 26578 raft_consensus.cc:2454] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 0.
I20260806 16:12:38.959259 26862 raft_consensus.cc:2454] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 0.
I20260806 16:12:38.959616 26652 leader_election.cc:302] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 50dca885423645b88348e9669a09968a; no voters:
I20260806 16:12:38.959863 26942 raft_consensus.cc:2781] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:38.959920 26942 raft_consensus.cc:482] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:38.959947 26942 raft_consensus.cc:3037] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.961014 26942 raft_consensus.cc:504] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.961364 26942 leader_election.cc:288] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 election: Requested vote from peers 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:38.961704 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81969afbb56a4502a8f202557f0db2c2" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f185700f7054edcbb145527ff14776b"
I20260806 16:12:38.961810 26578 raft_consensus.cc:3037] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.961795 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81969afbb56a4502a8f202557f0db2c2" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b"
I20260806 16:12:38.961951 26862 raft_consensus.cc:3037] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:38.962901 26578 raft_consensus.cc:2454] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 1.
I20260806 16:12:38.962904 26862 raft_consensus.cc:2454] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 1.
I20260806 16:12:38.963196 26652 leader_election.cc:302] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 50dca885423645b88348e9669a09968a; no voters:
I20260806 16:12:38.963493 26942 raft_consensus.cc:2781] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:38.963660 26942 raft_consensus.cc:686] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [term 1 LEADER]: Becoming Leader. State: Replica: 50dca885423645b88348e9669a09968a, State: Running, Role: LEADER
I20260806 16:12:38.963902 26942 consensus_queue.cc:237] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:38.965170 26982 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: TxnStatusTablet state changed. Reason: New leader 50dca885423645b88348e9669a09968a. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:38.965276 26982 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:38.965572 26984 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 16:12:38.965744 26984 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 16:12:38.966355 26418 catalog_manager.cc:5495] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a reported cstate change: term changed from 0 to 1, leader changed from <none> to 50dca885423645b88348e9669a09968a (127.18.90.2). New cstate: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
I20260806 16:12:39.026780 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.022626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.18.90.2:41305)
0806 16:12:39.022930 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.022937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.022949 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:39.023017 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.023019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.023071 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.023201 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:39.023484 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.024037 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.025015 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.025265 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.025388 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.025766 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.026333 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.026457 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.026531 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.026586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":143,"threads_started":1}
I20260806 16:12:39.026762 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.022165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:41305 (local address 127.0.0.1:34056)
0806 16:12:39.022323 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:39.022341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:39.022441 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:39.023225 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:39.023228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:39.023238 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:39.023371 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.023377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.024150 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.024156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.024889 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.024899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.025388 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.025392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.025503 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.025594 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.026427 (+ 833us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.026444 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:12:39.026521 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:12:39.026613 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 16:12:39.029824 26862 raft_consensus.cc:1261] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Refusing update from remote peer 50dca885423645b88348e9669a09968a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:39.030006 26578 raft_consensus.cc:1261] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Refusing update from remote peer 50dca885423645b88348e9669a09968a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:39.030326 26982 consensus_queue.cc:1035] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.030606 26942 consensus_queue.cc:1035] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.031916 26944 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: TxnStatusTablet state changed. Reason: New leader 50dca885423645b88348e9669a09968a. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:39.032029 26944 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:39.032378 26955 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: TxnStatusTablet state changed. Reason: New leader 50dca885423645b88348e9669a09968a. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:39.033182 26955 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:39.034852 26942 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:39.034960 26942 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:39.035808 26944 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:39.035915 26944 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 36b67ca47f594bf1bb6b87e72954355b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:39.036147 26955 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:39.036301 26955 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 9f185700f7054edcbb145527ff14776b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:12:39.036548 26982 tablet_replica.cc:439] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } } }
I20260806 16:12:39.036628 26982 tablet_replica.cc:442] T 81969afbb56a4502a8f202557f0db2c2 P 50dca885423645b88348e9669a09968a: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:12:39.051383 26955 consensus_queue.cc:1035] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.055186 26955 consensus_queue.cc:1035] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.056416 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.050378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:41305 (local address 127.18.90.1:49779)
0806 16:12:39.051221 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:39.051238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:39.051381 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:39.051729 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:39.051732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:39.051741 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:39.051864 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.051870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.052855 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.052858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.053769 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.053779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.054936 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.054941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.055065 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.055134 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.056063 (+ 929us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.056071 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:12:39.056136 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 16:12:39.056235 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":106,"threads_started":1}
I20260806 16:12:39.056950 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.050659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:49779 (local address 127.18.90.2:41305)
0806 16:12:39.050812 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.050817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.051305 (+ 488us) server_negotiation.cc:407] Connection header received
0806 16:12:39.051503 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.051507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.051571 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.051660 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:39.051981 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.052695 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.053909 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.054249 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.054350 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.055859 (+ 1509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.055953 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.056023 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.056764 (+ 741us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.056833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:39.082819 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.079093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:37273 (local address 127.18.90.1:32893)
0806 16:12:39.079245 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.079250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.079557 (+ 307us) server_negotiation.cc:407] Connection header received
0806 16:12:39.079688 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.079691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.079754 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.079836 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:39.080105 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.080703 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.081645 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.081886 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.082004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.082301 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.082380 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.082482 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.082607 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.082667 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:39.083289 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.078988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:32893 (local address 127.18.90.2:37273)
0806 16:12:39.079456 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:39.079471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:39.079601 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:39.079846 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:39.079849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:39.079856 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:39.080013 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.080019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.080837 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.080840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.081506 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.081514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.082016 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.082020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.082118 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.082172 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.082488 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.082497 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:12:39.083032 (+ 535us) client_negotiation.cc:240] Negotiation successful
0806 16:12:39.083114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
I20260806 16:12:39.107144 26418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:39.107331 26418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:39.109472 26418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Sending DeleteTablet for 3 replicas of tablet c9139acb98d64085afebe89a40ea90dd
I20260806 16:12:39.110123 26558 tablet_service.cc:1514] Processing DeleteTablet for tablet c9139acb98d64085afebe89a40ea90dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:53112
I20260806 16:12:39.110230 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet c9139acb98d64085afebe89a40ea90dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260806 16:12:39.110615 26701 tablet_service.cc:1514] Processing DeleteTablet for tablet c9139acb98d64085afebe89a40ea90dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34046
I20260806 16:12:39.110745 27008 tablet_replica.cc:330] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: stopping tablet replica
I20260806 16:12:39.110954 27008 raft_consensus.cc:2229] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:39.111052 27007 tablet_replica.cc:330] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: stopping tablet replica
I20260806 16:12:39.111148 27008 raft_consensus.cc:2258] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.111229 27007 raft_consensus.cc:2229] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:39.111454 27007 raft_consensus.cc:2258] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.112062 27008 ts_tablet_manager.cc:1905] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.112154 27007 ts_tablet_manager.cc:1905] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.114295 27009 tablet_replica.cc:330] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: stopping tablet replica
I20260806 16:12:39.114499 27009 raft_consensus.cc:2229] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:39.114666 27009 raft_consensus.cc:2258] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.115103 27007 ts_tablet_manager.cc:1918] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:39.115231 27007 log.cc:1196] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/c9139acb98d64085afebe89a40ea90dd
I20260806 16:12:39.115403 27009 ts_tablet_manager.cc:1905] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.115438 27008 ts_tablet_manager.cc:1918] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:39.115532 27008 log.cc:1196] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/c9139acb98d64085afebe89a40ea90dd
I20260806 16:12:39.115603 27007 ts_tablet_manager.cc:1939] T c9139acb98d64085afebe89a40ea90dd P 9f185700f7054edcbb145527ff14776b: Deleting consensus metadata
I20260806 16:12:39.115852 27008 ts_tablet_manager.cc:1939] T c9139acb98d64085afebe89a40ea90dd P 36b67ca47f594bf1bb6b87e72954355b: Deleting consensus metadata
I20260806 16:12:39.116400 26406 catalog_manager.cc:4841] TS 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893): tablet c9139acb98d64085afebe89a40ea90dd (table test-table [id=b59fc88764ef48be88d363fcd0570257]) successfully deleted
I20260806 16:12:39.116895 26406 catalog_manager.cc:4841] TS 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145): tablet c9139acb98d64085afebe89a40ea90dd (table test-table [id=b59fc88764ef48be88d363fcd0570257]) successfully deleted
I20260806 16:12:39.119343 27009 ts_tablet_manager.cc:1918] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:39.119413 27009 log.cc:1196] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/c9139acb98d64085afebe89a40ea90dd
I20260806 16:12:39.119669 27009 ts_tablet_manager.cc:1939] T c9139acb98d64085afebe89a40ea90dd P 50dca885423645b88348e9669a09968a: Deleting consensus metadata
I20260806 16:12:39.120348 26404 catalog_manager.cc:4841] TS 50dca885423645b88348e9669a09968a (127.18.90.2:41305): tablet c9139acb98d64085afebe89a40ea90dd (table test-table [id=b59fc88764ef48be88d363fcd0570257]) successfully deleted
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[26374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:12:39.148927 26974 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:39.142253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.18.90.62:45243)
0806 16:12:39.142398 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.142404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.142582 (+ 178us) server_negotiation.cc:407] Connection header received
0806 16:12:39.142771 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.142774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.142825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.142916 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:39.143639 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.144189 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.145173 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.145428 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.145536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.148810 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46832: BlockingRecv error: recv got EOF from 127.0.0.1:46832 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[26374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM
I20260806 16:12:39.158249 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.150348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.18.90.62:45243)
0806 16:12:39.150459 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.150462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.150654 (+ 192us) server_negotiation.cc:407] Connection header received
0806 16:12:39.150724 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.150726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.150769 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.150832 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:39.151387 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.151895 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.152723 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.152955 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.153071 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.155261 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:39.155265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:39.155267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:39.155299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:39.156608 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:39.157022 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:39.157026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:39.157027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:39.157117 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:39.157393 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:39.157397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:39.157399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:39.157575 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:39.157713 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.157954 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.158081 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:12:39.160704 26418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:39.161142 26418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:39.166028 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 97707f4f6e4a462cb1c8a4b5d168972f (DEFAULT_TABLE table=test-table [id=6a744f7f1bfc4d4d8ccbd025c06eee23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:39.166064 26701 tablet_service.cc:1467] Processing CreateTablet for tablet 97707f4f6e4a462cb1c8a4b5d168972f (DEFAULT_TABLE table=test-table [id=6a744f7f1bfc4d4d8ccbd025c06eee23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:39.166354 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97707f4f6e4a462cb1c8a4b5d168972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:39.166381 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97707f4f6e4a462cb1c8a4b5d168972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:39.168366 26936 tablet_bootstrap.cc:492] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Bootstrap starting.
I20260806 16:12:39.168581 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 97707f4f6e4a462cb1c8a4b5d168972f (DEFAULT_TABLE table=test-table [id=6a744f7f1bfc4d4d8ccbd025c06eee23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:39.168850 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97707f4f6e4a462cb1c8a4b5d168972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:39.169247 26936 tablet_bootstrap.cc:654] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:39.169837 26937 tablet_bootstrap.cc:492] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Bootstrap starting.
I20260806 16:12:39.170377 26936 tablet_bootstrap.cc:492] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: No bootstrap required, opened a new log
I20260806 16:12:39.170478 26936 ts_tablet_manager.cc:1397] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:39.170679 26937 tablet_bootstrap.cc:654] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:39.171007 26938 tablet_bootstrap.cc:492] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Bootstrap starting.
I20260806 16:12:39.170955 26936 raft_consensus.cc:348] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.171077 26936 raft_consensus.cc:374] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:39.171108 26936 raft_consensus.cc:729] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dca885423645b88348e9669a09968a, State: Initialized, Role: FOLLOWER
I20260806 16:12:39.171250 26936 consensus_queue.cc:260] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.171556 26936 ts_tablet_manager.cc:1428] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:39.171855 26937 tablet_bootstrap.cc:492] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: No bootstrap required, opened a new log
I20260806 16:12:39.171888 26938 tablet_bootstrap.cc:654] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:39.171936 26937 ts_tablet_manager.cc:1397] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:39.172375 26937 raft_consensus.cc:348] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.172497 26937 raft_consensus.cc:374] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:39.172537 26937 raft_consensus.cc:729] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Initialized, Role: FOLLOWER
I20260806 16:12:39.172684 26937 consensus_queue.cc:260] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.172978 26937 ts_tablet_manager.cc:1428] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:39.174067 26938 tablet_bootstrap.cc:492] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: No bootstrap required, opened a new log
I20260806 16:12:39.174154 26938 ts_tablet_manager.cc:1397] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 16:12:39.174971 26938 raft_consensus.cc:348] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.175086 26938 raft_consensus.cc:374] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:39.175132 26938 raft_consensus.cc:729] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b67ca47f594bf1bb6b87e72954355b, State: Initialized, Role: FOLLOWER
I20260806 16:12:39.175269 26938 consensus_queue.cc:260] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.175565 26938 ts_tablet_manager.cc:1428] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 16:12:39.388163 26944 raft_consensus.cc:482] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:39.388352 26944 raft_consensus.cc:504] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.389444 26944 leader_election.cc:288] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893), 50dca885423645b88348e9669a09968a (127.18.90.2:41305)
I20260806 16:12:39.393548 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.389806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:44989 (local address 127.18.90.2:41305)
0806 16:12:39.389943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.389947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.390234 (+ 287us) server_negotiation.cc:407] Connection header received
0806 16:12:39.390354 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.390356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.390401 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.390500 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:39.390824 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.391320 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.392581 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.392963 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.393089 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.393156 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.393357 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.393433 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:12:39.393548 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.389778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:35189 (local address 127.18.90.1:32893)
0806 16:12:39.389919 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.389925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.390132 (+ 207us) server_negotiation.cc:407] Connection header received
0806 16:12:39.390309 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.390312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.390371 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.390431 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:12:39.390795 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.391334 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.392540 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.392945 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.393028 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.393125 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.393344 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.393422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:12:39.393586 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.389706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:41305 (local address 127.18.90.3:44989)
0806 16:12:39.390078 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:39.390115 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:39.390243 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:39.390539 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:39.390544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:39.390572 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:39.390684 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.390688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.391464 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.391467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.392431 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.392441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.393110 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.393114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.393222 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.393259 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 16:12:39.393338 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:12:39.393414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":59,"thread_start_us":119,"threads_started":1}
I20260806 16:12:39.393611 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.389692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:32893 (local address 127.18.90.3:35189)
0806 16:12:39.390029 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:39.390042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:39.390142 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:39.390471 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:39.390475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:39.390496 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:12:39.390669 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.390675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.391476 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.391479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.392420 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.392430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.393073 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.393077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.393179 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.393214 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 16:12:39.393320 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 16:12:39.393396 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260806 16:12:39.394101 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97707f4f6e4a462cb1c8a4b5d168972f" candidate_uuid: "36b67ca47f594bf1bb6b87e72954355b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f185700f7054edcbb145527ff14776b" is_pre_election: true
I20260806 16:12:39.394145 26721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97707f4f6e4a462cb1c8a4b5d168972f" candidate_uuid: "36b67ca47f594bf1bb6b87e72954355b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50dca885423645b88348e9669a09968a" is_pre_election: true
I20260806 16:12:39.394308 26578 raft_consensus.cc:2454] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b67ca47f594bf1bb6b87e72954355b in term 0.
I20260806 16:12:39.394307 26721 raft_consensus.cc:2454] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b67ca47f594bf1bb6b87e72954355b in term 0.
I20260806 16:12:39.394733 26794 leader_election.cc:302] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 50dca885423645b88348e9669a09968a; no voters:
I20260806 16:12:39.395058 26944 raft_consensus.cc:2781] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:39.395116 26944 raft_consensus.cc:482] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:39.395145 26944 raft_consensus.cc:3037] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:39.396202 26944 raft_consensus.cc:504] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.396665 26944 leader_election.cc:288] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [CANDIDATE]: Term 1 election: Requested vote from peers 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893), 50dca885423645b88348e9669a09968a (127.18.90.2:41305)
I20260806 16:12:39.397037 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97707f4f6e4a462cb1c8a4b5d168972f" candidate_uuid: "36b67ca47f594bf1bb6b87e72954355b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f185700f7054edcbb145527ff14776b"
I20260806 16:12:39.397084 26721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97707f4f6e4a462cb1c8a4b5d168972f" candidate_uuid: "36b67ca47f594bf1bb6b87e72954355b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50dca885423645b88348e9669a09968a"
I20260806 16:12:39.397161 26578 raft_consensus.cc:3037] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:39.397238 26721 raft_consensus.cc:3037] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:39.398105 26578 raft_consensus.cc:2454] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b67ca47f594bf1bb6b87e72954355b in term 1.
I20260806 16:12:39.398195 26721 raft_consensus.cc:2454] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b67ca47f594bf1bb6b87e72954355b in term 1.
I20260806 16:12:39.398464 26796 leader_election.cc:302] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 9f185700f7054edcbb145527ff14776b; no voters:
I20260806 16:12:39.398692 26944 raft_consensus.cc:2781] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:39.398878 26944 raft_consensus.cc:686] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 1 LEADER]: Becoming Leader. State: Replica: 36b67ca47f594bf1bb6b87e72954355b, State: Running, Role: LEADER
I20260806 16:12:39.399128 26944 consensus_queue.cc:237] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.401522 26416 catalog_manager.cc:5495] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3). New cstate: current_term: 1 leader_uuid: "36b67ca47f594bf1bb6b87e72954355b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } health_report { overall_health: HEALTHY } } }
I20260806 16:12:39.414436 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.411107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.18.90.3:44145)
0806 16:12:39.411232 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.411236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.411248 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:39.411401 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.411403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.411453 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.411537 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:39.411747 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.412228 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.413144 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.413354 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.413429 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.413654 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.414085 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.414171 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.414230 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.414276 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:12:39.416980 26721 raft_consensus.cc:1261] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Refusing update from remote peer 36b67ca47f594bf1bb6b87e72954355b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:39.417019 26578 raft_consensus.cc:1261] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Refusing update from remote peer 36b67ca47f594bf1bb6b87e72954355b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:39.417598 27027 consensus_queue.cc:1035] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.417768 26944 consensus_queue.cc:1035] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.421412 26992 mvcc.cc:205] Tried to move back new op lower bound from 7315590182566883328 to 7315590182501281792. Current Snapshot: MvccSnapshot[applied={T|T < 7315590182566883328}]
I20260806 16:12:39.428475 26416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:39.428627 26416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:39.430814 26416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Sending DeleteTablet for 3 replicas of tablet 97707f4f6e4a462cb1c8a4b5d168972f
I20260806 16:12:39.431351 26558 tablet_service.cc:1514] Processing DeleteTablet for tablet 97707f4f6e4a462cb1c8a4b5d168972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:53112
I20260806 16:12:39.431461 26701 tablet_service.cc:1514] Processing DeleteTablet for tablet 97707f4f6e4a462cb1c8a4b5d168972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34046
I20260806 16:12:39.431553 27007 tablet_replica.cc:330] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: stopping tablet replica
I20260806 16:12:39.431583 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet 97707f4f6e4a462cb1c8a4b5d168972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260806 16:12:39.431703 27007 raft_consensus.cc:2229] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:39.431758 27008 tablet_replica.cc:330] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: stopping tablet replica
I20260806 16:12:39.431807 27009 tablet_replica.cc:330] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: stopping tablet replica
I20260806 16:12:39.431843 27007 raft_consensus.cc:2258] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.431859 27008 raft_consensus.cc:2229] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:39.431943 27009 raft_consensus.cc:2229] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:39.432081 27009 raft_consensus.cc:2258] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.432123 27008 raft_consensus.cc:2258] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.432793 27007 ts_tablet_manager.cc:1905] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.433020 27009 ts_tablet_manager.cc:1905] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.433581 27008 ts_tablet_manager.cc:1905] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.434998 27007 ts_tablet_manager.cc:1918] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:39.435099 27007 log.cc:1196] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/97707f4f6e4a462cb1c8a4b5d168972f
I20260806 16:12:39.435356 27007 ts_tablet_manager.cc:1939] T 97707f4f6e4a462cb1c8a4b5d168972f P 9f185700f7054edcbb145527ff14776b: Deleting consensus metadata
I20260806 16:12:39.435555 27009 ts_tablet_manager.cc:1918] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:39.435638 27009 log.cc:1196] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/97707f4f6e4a462cb1c8a4b5d168972f
I20260806 16:12:39.435906 27009 ts_tablet_manager.cc:1939] T 97707f4f6e4a462cb1c8a4b5d168972f P 50dca885423645b88348e9669a09968a: Deleting consensus metadata
I20260806 16:12:39.436043 26406 catalog_manager.cc:4841] TS 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893): tablet 97707f4f6e4a462cb1c8a4b5d168972f (table test-table [id=6a744f7f1bfc4d4d8ccbd025c06eee23]) successfully deleted
I20260806 16:12:39.436154 27008 ts_tablet_manager.cc:1918] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:12:39.436246 27008 log.cc:1196] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/97707f4f6e4a462cb1c8a4b5d168972f
I20260806 16:12:39.436543 27008 ts_tablet_manager.cc:1939] T 97707f4f6e4a462cb1c8a4b5d168972f P 36b67ca47f594bf1bb6b87e72954355b: Deleting consensus metadata
I20260806 16:12:39.436632 26404 catalog_manager.cc:4841] TS 50dca885423645b88348e9669a09968a (127.18.90.2:41305): tablet 97707f4f6e4a462cb1c8a4b5d168972f (table test-table [id=6a744f7f1bfc4d4d8ccbd025c06eee23]) successfully deleted
I20260806 16:12:39.437156 26406 catalog_manager.cc:4841] TS 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145): tablet 97707f4f6e4a462cb1c8a4b5d168972f (table test-table [id=6a744f7f1bfc4d4d8ccbd025c06eee23]) successfully deleted
I20260806 16:12:39.437196 26416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:12:39.437608 26416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:12:39.442027 26558 tablet_service.cc:1467] Processing CreateTablet for tablet 5d86940279244949a37fa6142fb7407d (DEFAULT_TABLE table=test-table [id=692df2aae4344031814124a8a3ca1250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:39.442169 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 5d86940279244949a37fa6142fb7407d (DEFAULT_TABLE table=test-table [id=692df2aae4344031814124a8a3ca1250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:39.442188 26701 tablet_service.cc:1467] Processing CreateTablet for tablet 5d86940279244949a37fa6142fb7407d (DEFAULT_TABLE table=test-table [id=692df2aae4344031814124a8a3ca1250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:12:39.442303 26558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d86940279244949a37fa6142fb7407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:39.442412 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d86940279244949a37fa6142fb7407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:39.442412 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d86940279244949a37fa6142fb7407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:39.444672 26937 tablet_bootstrap.cc:492] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Bootstrap starting.
I20260806 16:12:39.445001 26938 tablet_bootstrap.cc:492] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Bootstrap starting.
I20260806 16:12:39.445528 26937 tablet_bootstrap.cc:654] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:39.445928 26938 tablet_bootstrap.cc:654] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:39.446081 26936 tablet_bootstrap.cc:492] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Bootstrap starting.
I20260806 16:12:39.446947 26936 tablet_bootstrap.cc:654] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:39.447417 26938 tablet_bootstrap.cc:492] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: No bootstrap required, opened a new log
I20260806 16:12:39.447516 26938 ts_tablet_manager.cc:1397] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 16:12:39.448024 26938 raft_consensus.cc:348] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.448156 26938 raft_consensus.cc:374] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:39.448181 26936 tablet_bootstrap.cc:492] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: No bootstrap required, opened a new log
I20260806 16:12:39.448194 26938 raft_consensus.cc:729] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b67ca47f594bf1bb6b87e72954355b, State: Initialized, Role: FOLLOWER
I20260806 16:12:39.448292 26936 ts_tablet_manager.cc:1397] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:39.448333 26938 consensus_queue.cc:260] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.448447 26937 tablet_bootstrap.cc:492] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: No bootstrap required, opened a new log
I20260806 16:12:39.448529 26937 ts_tablet_manager.cc:1397] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 16:12:39.448632 26938 ts_tablet_manager.cc:1428] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:39.448660 26936 raft_consensus.cc:348] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.448756 26936 raft_consensus.cc:374] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:39.448786 26936 raft_consensus.cc:729] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dca885423645b88348e9669a09968a, State: Initialized, Role: FOLLOWER
I20260806 16:12:39.448899 26936 consensus_queue.cc:260] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.448938 26937 raft_consensus.cc:348] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.449045 26937 raft_consensus.cc:374] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:39.449076 26937 raft_consensus.cc:729] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f185700f7054edcbb145527ff14776b, State: Initialized, Role: FOLLOWER
I20260806 16:12:39.449141 26936 ts_tablet_manager.cc:1428] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:12:39.449198 26937 consensus_queue.cc:260] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.449452 26937 ts_tablet_manager.cc:1428] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:12:39.458997 26994 raft_consensus.cc:482] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:12:39.459089 26994 raft_consensus.cc:504] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.459527 26994 leader_election.cc:288] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:39.459899 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d86940279244949a37fa6142fb7407d" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b" is_pre_election: true
I20260806 16:12:39.459905 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d86940279244949a37fa6142fb7407d" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f185700f7054edcbb145527ff14776b" is_pre_election: true
I20260806 16:12:39.460040 26578 raft_consensus.cc:2454] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 0.
I20260806 16:12:39.460040 26862 raft_consensus.cc:2454] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 0.
I20260806 16:12:39.460361 26652 leader_election.cc:302] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b67ca47f594bf1bb6b87e72954355b, 50dca885423645b88348e9669a09968a; no voters:
I20260806 16:12:39.460554 26994 raft_consensus.cc:2781] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:12:39.460628 26994 raft_consensus.cc:482] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:12:39.460690 26994 raft_consensus.cc:3037] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:39.461892 26994 raft_consensus.cc:504] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.462229 26994 leader_election.cc:288] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 election: Requested vote from peers 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893), 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145)
I20260806 16:12:39.462586 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d86940279244949a37fa6142fb7407d" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f185700f7054edcbb145527ff14776b"
I20260806 16:12:39.462636 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d86940279244949a37fa6142fb7407d" candidate_uuid: "50dca885423645b88348e9669a09968a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b67ca47f594bf1bb6b87e72954355b"
I20260806 16:12:39.462739 26862 raft_consensus.cc:3037] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:39.462739 26578 raft_consensus.cc:3037] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:39.463847 26862 raft_consensus.cc:2454] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 1.
I20260806 16:12:39.463863 26578 raft_consensus.cc:2454] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50dca885423645b88348e9669a09968a in term 1.
I20260806 16:12:39.464159 26652 leader_election.cc:302] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50dca885423645b88348e9669a09968a, 9f185700f7054edcbb145527ff14776b; no voters:
I20260806 16:12:39.464389 26994 raft_consensus.cc:2781] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:39.464519 26994 raft_consensus.cc:686] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 1 LEADER]: Becoming Leader. State: Replica: 50dca885423645b88348e9669a09968a, State: Running, Role: LEADER
I20260806 16:12:39.464675 26994 consensus_queue.cc:237] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } }
I20260806 16:12:39.466315 26416 catalog_manager.cc:5495] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a reported cstate change: term changed from 0 to 1, leader changed from <none> to 50dca885423645b88348e9669a09968a (127.18.90.2). New cstate: current_term: 1 leader_uuid: "50dca885423645b88348e9669a09968a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50dca885423645b88348e9669a09968a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 41305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 } health_report { overall_health: UNKNOWN } } }
I20260806 16:12:39.484023 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.480528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.18.90.2:41305)
0806 16:12:39.480641 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.480645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.480657 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:39.480787 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.480789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.480827 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.480887 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:39.481146 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.481649 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.482870 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.483083 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.483160 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.483490 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.483579 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.483652 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.483834 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.483889 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:12:39.492735 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.489557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:52537 (local address 127.18.90.2:41305)
0806 16:12:39.489663 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:39.489666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:39.489686 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:12:39.489804 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:39.489806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:39.489844 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:39.489914 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:12:39.490139 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.490601 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.491753 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.491951 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.492028 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.492337 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.492405 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.492463 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:39.492577 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 16:12:39.492615 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:12:39.492831 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:39.489480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:41305 (local address 127.18.90.2:52537)
0806 16:12:39.489635 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:39.489648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:39.489715 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:39.489946 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:39.489949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:39.489956 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:12:39.490044 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.490050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.490715 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.490718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.491633 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:39.491644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:39.492056 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:39.492060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:39.492144 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:39.492173 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:12:39.492491 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:12:39.492498 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:12:39.492573 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:12:39.492654 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 16:12:39.498850 26577 raft_consensus.cc:1261] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Refusing update from remote peer 50dca885423645b88348e9669a09968a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:39.498849 26862 raft_consensus.cc:1261] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Refusing update from remote peer 50dca885423645b88348e9669a09968a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:12:39.499238 26982 consensus_queue.cc:1035] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f185700f7054edcbb145527ff14776b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.499382 26994 consensus_queue.cc:1035] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b67ca47f594bf1bb6b87e72954355b" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:12:39.535192 26416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:39.535357 26416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:12:39.537405 26416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a12ff0590be54fc4880f6ecff0b8d83c: Sending DeleteTablet for 3 replicas of tablet 5d86940279244949a37fa6142fb7407d
I20260806 16:12:39.537909 26558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d86940279244949a37fa6142fb7407d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:53112
I20260806 16:12:39.537909 26700 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d86940279244949a37fa6142fb7407d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34046
I20260806 16:12:39.538074 27007 tablet_replica.cc:330] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: stopping tablet replica
I20260806 16:12:39.538074 27009 tablet_replica.cc:330] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: stopping tablet replica
I20260806 16:12:39.538178 27007 raft_consensus.cc:2229] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:39.538188 27009 raft_consensus.cc:2229] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:12:39.538288 27007 raft_consensus.cc:2258] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.538458 27009 raft_consensus.cc:2258] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.539134 27007 ts_tablet_manager.cc:1905] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.541309 27007 ts_tablet_manager.cc:1918] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:39.541402 27007 log.cc:1196] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/5d86940279244949a37fa6142fb7407d
I20260806 16:12:39.541656 27007 ts_tablet_manager.cc:1939] T 5d86940279244949a37fa6142fb7407d P 9f185700f7054edcbb145527ff14776b: Deleting consensus metadata
I20260806 16:12:39.542294 26406 catalog_manager.cc:4841] TS 9f185700f7054edcbb145527ff14776b (127.18.90.1:32893): tablet 5d86940279244949a37fa6142fb7407d (table test-table [id=692df2aae4344031814124a8a3ca1250]) successfully deleted
I20260806 16:12:39.542587 27009 ts_tablet_manager.cc:1905] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.544468 27009 ts_tablet_manager.cc:1918] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:39.544559 27009 log.cc:1196] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/5d86940279244949a37fa6142fb7407d
I20260806 16:12:39.544788 27009 ts_tablet_manager.cc:1939] T 5d86940279244949a37fa6142fb7407d P 50dca885423645b88348e9669a09968a: Deleting consensus metadata
I20260806 16:12:39.545444 26404 catalog_manager.cc:4841] TS 50dca885423645b88348e9669a09968a (127.18.90.2:41305): tablet 5d86940279244949a37fa6142fb7407d (table test-table [id=692df2aae4344031814124a8a3ca1250]) successfully deleted
I20260806 16:12:39.546015 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d86940279244949a37fa6142fb7407d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:12:39 UTC) from {username='oryx', principal='oryx/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260806 16:12:39.546270 27008 tablet_replica.cc:330] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: stopping tablet replica
I20260806 16:12:39.546409 27008 raft_consensus.cc:2229] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:12:39.546543 27008 raft_consensus.cc:2258] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:12:39.547209 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 26492
I20260806 16:12:39.547473 27008 ts_tablet_manager.cc:1905] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:12:39.550493 27008 ts_tablet_manager.cc:1918] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:12:39.550571 27008 log.cc:1196] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/5d86940279244949a37fa6142fb7407d
I20260806 16:12:39.550829 27008 ts_tablet_manager.cc:1939] T 5d86940279244949a37fa6142fb7407d P 36b67ca47f594bf1bb6b87e72954355b: Deleting consensus metadata
I20260806 16:12:39.551729 26406 catalog_manager.cc:4841] TS 36b67ca47f594bf1bb6b87e72954355b (127.18.90.3:44145): tablet 5d86940279244949a37fa6142fb7407d (table test-table [id=692df2aae4344031814124a8a3ca1250]) successfully deleted
I20260806 16:12:39.559142 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 26633
I20260806 16:12:39.568378 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 26776
I20260806 16:12:39.575973 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 26387
2026-08-06T16:12:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3614 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[27044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[27044](info): set up 2 sockets
Aug 06 16:12:39 dist-test-slave-0b58 krb5kdc[27044](info): commencing operation
krb5kdc: starting...
W20260806 16:12:40.613915 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:40 dist-test-slave-0b58 krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:40.747203 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:35875
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:35875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:40.862860 27057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:40.863152 27057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:40.863195 27057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:40.867420 27057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:40.867494 27057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:40.867514 27057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:40.867532 27057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:40.867547 27057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:40.873026 27057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:35875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:40.874205 27057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:40.875195 27057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:40.883860 27064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:40.883860 27063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:40.883874 27066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:40.884357 27057 server_base.cc:1034] running on GCE node
I20260806 16:12:40.884980 27057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:40.886147 27057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:40.887348 27057 hybrid_clock.cc:648] HybridClock initialized: now 1786032760887338 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T16:12:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1318 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:12:40 dist-test-slave-0b58 krb5kdc[27071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:12:40 dist-test-slave-0b58 krb5kdc[27071](info): set up 2 sockets
Aug 06 16:12:40 dist-test-slave-0b58 krb5kdc[27071](info): commencing operation
krb5kdc: starting...
W20260806 16:12:41.942075 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:12:42.071571 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33657
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:42.184474 27085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:42.184746 27085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:42.184799 27085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:42.188683 27085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:42.188740 27085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:42.188760 27085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:42.188777 27085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:42.188793 27085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:42.193944 27085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:42.194981 27085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:42.195989 27085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:42.204547 27090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.204558 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.204555 27093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:42.204594 27085 server_base.cc:1034] running on GCE node
I20260806 16:12:42.205348 27085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:42.206506 27085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:42.207670 27085 hybrid_clock.cc:648] HybridClock initialized: now 1786032762207641 us; error 42 us; skew 500 ppm
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:42.210474 27085 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:42.211436 27085 webserver.cc:460] Webserver started at http://127.18.90.62:41151/ using document root <none> and password file <none>
I20260806 16:12:42.212029 27085 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:42.212091 27085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:42.212383 27085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:42.214450 27085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "94b278c8ba724628903b5dbe202ddc64"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "c86ac85b9e8b254483c62b22586eea1f"
server_key_iv: "c760d43d65c9cebddb6a6f31ebd90103"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.215075 27085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "94b278c8ba724628903b5dbe202ddc64"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "c86ac85b9e8b254483c62b22586eea1f"
server_key_iv: "c760d43d65c9cebddb6a6f31ebd90103"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.219009 27085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:12:42.221592 27100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.223140 27085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:12:42.223280 27085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "94b278c8ba724628903b5dbe202ddc64"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "c86ac85b9e8b254483c62b22586eea1f"
server_key_iv: "c760d43d65c9cebddb6a6f31ebd90103"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.223418 27085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:42.240895 27085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:42.243983 27085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:42.244151 27085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:42.251470 27085 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:33657
I20260806 16:12:42.251485 27152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:33657 every 8 connection(s)
I20260806 16:12:42.252506 27085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:42.255359 27153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:42.258490 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27085
I20260806 16:12:42.258639 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:42.258961 18792 external_mini_cluster.cc:1428] Setting key e240e271b4a10f6ea9ec01087244c035
I20260806 16:12:42.261816 27153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Bootstrap starting.
I20260806 16:12:42.264055 27153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:42.264871 27153 log.cc:824] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:42.267279 27153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: No bootstrap required, opened a new log
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:42.271111 27153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:42.271360 27153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:42.271396 27153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b278c8ba724628903b5dbe202ddc64, State: Initialized, Role: FOLLOWER
I20260806 16:12:42.271930 27153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:42.272111 27153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:42.272198 27153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:42.272341 27153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:42.273458 27153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:42.273854 27153 leader_election.cc:302] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b278c8ba724628903b5dbe202ddc64; no voters:
I20260806 16:12:42.274245 27153 leader_election.cc:288] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:42.274314 27158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:42.274521 27158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 LEADER]: Becoming Leader. State: Replica: 94b278c8ba724628903b5dbe202ddc64, State: Running, Role: LEADER
I20260806 16:12:42.274881 27158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:42.275270 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.260506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.18.90.62:33657)
0806 16:12:42.261008 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:42.261024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:42.261061 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:12:42.261698 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:42.261716 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:42.262119 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:42.262474 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:42.263315 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.264469 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.265490 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.266219 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.266389 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.269205 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:42.269211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:42.269226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:42.269253 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:42.271657 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.272214 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.272222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.272229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.272411 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.272789 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.272792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.272793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.273224 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:42.273390 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:42.273765 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 16:12:42.274122 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":354,"thread_start_us":175,"threads_started":1}
I20260806 16:12:42.275616 27153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:42.276093 27158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b278c8ba724628903b5dbe202ddc64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } } }
I20260806 16:12:42.276206 27158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:42.276623 27162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:42.276583 27159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b278c8ba724628903b5dbe202ddc64. Latest consensus state: current_term: 1 leader_uuid: "94b278c8ba724628903b5dbe202ddc64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } } }
I20260806 16:12:42.276679 27159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:42.279930 27162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:42.286257 27162 catalog_manager.cc:1347] Generated new cluster ID: f2c162c9915b4454a69b79ca15b0c776
I20260806 16:12:42.286338 27162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:42.305037 27162 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:42.306221 27162 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:42.318328 27162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Generated new TSK 0
I20260806 16:12:42.319125 27162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:12:42.352716 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33657
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:42.463712 27179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:42.463984 27179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:42.464025 27179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:42.467926 27179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:42.467983 27179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:42.468072 27179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:42.473258 27179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:42.474238 27179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:42.475238 27179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:42.484771 27184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.484766 27187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:42.484895 27179 server_base.cc:1034] running on GCE node
W20260806 16:12:42.484804 27185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:42.485455 27179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:42.486102 27179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:42.487324 27179 hybrid_clock.cc:648] HybridClock initialized: now 1786032762487306 us; error 28 us; skew 500 ppm
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:42.490125 27179 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:12:42.491005 27179 webserver.cc:460] Webserver started at http://127.18.90.1:35943/ using document root <none> and password file <none>
I20260806 16:12:42.491591 27179 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:42.491707 27179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:42.491964 27179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:42.494053 27179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "0ab3e99ca28b42efb2e7af5c1dca71da"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "06ec6593778c08699edac6e4096e6417"
server_key_iv: "7e21e38921f9013ed816059e9f878713"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.494632 27179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "0ab3e99ca28b42efb2e7af5c1dca71da"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "06ec6593778c08699edac6e4096e6417"
server_key_iv: "7e21e38921f9013ed816059e9f878713"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.498364 27179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:12:42.500851 27194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.502018 27179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:42.502157 27179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "0ab3e99ca28b42efb2e7af5c1dca71da"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "06ec6593778c08699edac6e4096e6417"
server_key_iv: "7e21e38921f9013ed816059e9f878713"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.502271 27179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:42.517053 27179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:42.520335 27179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:42.520552 27179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:42.521216 27179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:42.522022 27179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:42.522070 27179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.522110 27179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:42.522131 27179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.530989 27179 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:33661
I20260806 16:12:42.531009 27309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:33661 every 8 connection(s)
I20260806 16:12:42.531894 27179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:42.539405 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27179
I20260806 16:12:42.539553 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:42.539906 18792 external_mini_cluster.cc:1428] Setting key 2cc64fb95da62243b4f0ecce23444e3d
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:42.545594 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.533745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:44851 (local address 127.18.90.62:33657)
0806 16:12:42.533887 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:42.533891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:42.534708 (+ 817us) server_negotiation.cc:407] Connection header received
0806 16:12:42.535921 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:42.535924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:42.535971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:42.536046 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:42.537229 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.537761 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.538843 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.539077 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.539154 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.542148 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:42.542151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:42.542153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:42.542176 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:42.543548 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.544044 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.544048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.544050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.544129 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.544508 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.544512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.544514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.544695 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:42.544808 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:42.545305 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 16:12:42.545429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:42.546509 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.534133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:44851)
0806 16:12:42.534560 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:42.534601 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:42.535727 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:42.536202 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:42.536209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:42.536620 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:42.537082 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:42.537094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.537870 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.537872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:42.538743 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:42.538751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.539249 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.539256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:42.539469 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.540188 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:42.540208 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:42.541977 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:42.543666 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:42.543672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:42.543686 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:42.543941 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:42.544221 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:42.544225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:42.544227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:42.544411 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:42.544809 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:42.544817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:42.545064 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 16:12:42.545250 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 16:12:42.545575 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260806 16:12:42.547976 27310 heartbeater.cc:347] Connected to a master server at 127.18.90.62:33657
I20260806 16:12:42.548225 27310 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:42.548738 27310 heartbeater.cc:511] Master 127.18.90.62:33657 requested a full tablet report, sending...
I20260806 16:12:42.550485 27117 ts_manager.cc:194] Registered new tserver with Master: 0ab3e99ca28b42efb2e7af5c1dca71da (127.18.90.1:33661)
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:12:42.551728 27117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:44851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:12:42.569499 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33657
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:42.681406 27315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:42.681660 27315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:42.681704 27315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:42.685627 27315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:42.685683 27315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:42.685741 27315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:42.690887 27315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:42.691931 27315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:42.692978 27315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:42.702486 27321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.702486 27323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.702497 27320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:42.702565 27315 server_base.cc:1034] running on GCE node
I20260806 16:12:42.703177 27315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:42.703809 27315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:42.704977 27315 hybrid_clock.cc:648] HybridClock initialized: now 1786032762704950 us; error 24 us; skew 500 ppm
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:42.707777 27315 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:12:42.708734 27315 webserver.cc:460] Webserver started at http://127.18.90.2:38145/ using document root <none> and password file <none>
I20260806 16:12:42.709295 27315 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:42.709353 27315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:42.709584 27315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:42.711576 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "ee616142b7b7466291838bcf93571169"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "76dd6cec45db5f972ac653d6e4038301"
server_key_iv: "c9eac7cefd7f7a85ed3c4bc339fe6fba"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.712136 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "ee616142b7b7466291838bcf93571169"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "76dd6cec45db5f972ac653d6e4038301"
server_key_iv: "c9eac7cefd7f7a85ed3c4bc339fe6fba"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.715735 27315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:12:42.718124 27331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.719399 27315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:42.719532 27315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "ee616142b7b7466291838bcf93571169"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "76dd6cec45db5f972ac653d6e4038301"
server_key_iv: "c9eac7cefd7f7a85ed3c4bc339fe6fba"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.719640 27315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:42.748108 27315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:42.751600 27315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:42.751820 27315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:42.752607 27315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:42.753614 27315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:42.753681 27315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.753746 27315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:42.753772 27315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.762518 27315 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44201
I20260806 16:12:42.762545 27445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44201 every 8 connection(s)
I20260806 16:12:42.763445 27315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:42.766470 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27315
I20260806 16:12:42.766618 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:42.767012 18792 external_mini_cluster.cc:1428] Setting key 5cf746c66ff175bd00ec79fcce29a92b
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:12:42.778752 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.765377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:51765 (local address 127.18.90.62:33657)
0806 16:12:42.765533 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:42.765536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:42.766464 (+ 928us) server_negotiation.cc:407] Connection header received
0806 16:12:42.768120 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:42.768123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:42.768196 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:42.768316 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:42.769903 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.770431 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.771727 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.772138 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.772290 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.774814 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:42.774820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:42.774826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:42.774865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:42.776640 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.777221 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.777228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.777231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.777330 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.777679 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.777684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.777688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.777920 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:42.778026 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:42.778527 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 16:12:42.778625 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:12:42.779711 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.765710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:51765)
0806 16:12:42.766255 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:42.766319 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:42.767911 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:42.768513 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:42.768529 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:42.769019 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:42.769736 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:42.769760 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.770556 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.770562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:42.771570 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:42.771580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.772358 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.772362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:42.772508 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.773142 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:42.773157 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:42.774635 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:42.776767 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:42.776772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:42.776782 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:42.777112 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:42.777440 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:42.777443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:42.777444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:42.777571 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:42.778020 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:42.778025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:42.778294 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 16:12:42.778444 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 16:12:42.778682 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:12:42.781026 27446 heartbeater.cc:347] Connected to a master server at 127.18.90.62:33657
I20260806 16:12:42.781277 27446 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:42.781756 27446 heartbeater.cc:511] Master 127.18.90.62:33657 requested a full tablet report, sending...
I20260806 16:12:42.782877 27117 ts_manager.cc:194] Registered new tserver with Master: ee616142b7b7466291838bcf93571169 (127.18.90.2:44201)
I20260806 16:12:42.783471 27117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:51765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:12:42.795113 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33657
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:42.905478 27451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:42.905764 27451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:42.905804 27451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:42.910027 27451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:42.910089 27451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:42.910146 27451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:42.915093 27451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:42.915982 27451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:42.916908 27451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:42.926060 27459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.926034 27456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:42.926051 27457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:42.926185 27451 server_base.cc:1034] running on GCE node
I20260806 16:12:42.926713 27451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:42.927279 27451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:42.928469 27451 hybrid_clock.cc:648] HybridClock initialized: now 1786032762928457 us; error 23 us; skew 500 ppm
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:42.931068 27451 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:12:42.931910 27451 webserver.cc:460] Webserver started at http://127.18.90.3:35321/ using document root <none> and password file <none>
I20260806 16:12:42.932492 27451 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:42.932590 27451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:42.932808 27451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:42.934728 27451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "c0507b14af5749dfbe9831472fa18835"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "3eb5576ab561cf58d747ed55eef17ee6"
server_key_iv: "b8ba9ca0fa12db505fe759f74915077f"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.935251 27451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "c0507b14af5749dfbe9831472fa18835"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "3eb5576ab561cf58d747ed55eef17ee6"
server_key_iv: "b8ba9ca0fa12db505fe759f74915077f"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.938690 27451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:12:42.941039 27466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.942193 27451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:42.942286 27451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "c0507b14af5749dfbe9831472fa18835"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "3eb5576ab561cf58d747ed55eef17ee6"
server_key_iv: "b8ba9ca0fa12db505fe759f74915077f"
server_key_version: "encryptionkey@0"
I20260806 16:12:42.942360 27451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:42.968005 27451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:42.970927 27451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:42.971078 27451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:42.971638 27451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:42.972528 27451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:42.972575 27451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.972615 27451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:42.972632 27451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:42.981777 27451 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:33117
I20260806 16:12:42.981851 27581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:33117 every 8 connection(s)
I20260806 16:12:42.982690 27451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
Aug 06 16:12:42 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032762, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:12:42.992146 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27451
I20260806 16:12:42.992336 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:42.992683 18792 external_mini_cluster.cc:1428] Setting key 149f7d409f4be572fd6dc77fc4db54cc
I20260806 16:12:42.996194 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.984463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:40979 (local address 127.18.90.62:33657)
0806 16:12:42.984599 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:42.984602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:42.985415 (+ 813us) server_negotiation.cc:407] Connection header received
0806 16:12:42.986732 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:42.986736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:42.986795 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:42.986903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:42.988030 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.988617 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.989756 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.989977 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.990093 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.992626 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:42.992630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:42.992633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:42.992661 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:42.994319 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.994915 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.994918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.994920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.995031 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:42.995335 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:42.995337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:42.995338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:42.995499 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:42.995575 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:42.995994 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 16:12:42.996082 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:12:42.997041 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:42.984846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:40979)
0806 16:12:42.985270 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:42.985311 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:42.986523 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:42.987074 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:42.987081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:42.987417 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:42.987881 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:42.987894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.988743 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.988746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:42.989624 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:42.989633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:42.990131 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:42.990135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:42.990304 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:42.990899 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:42.990915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:42.992455 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:42.994426 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:42.994431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:42.994444 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:42.994773 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:42.995122 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:12:42.995125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:12:42.995126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:12:42.995251 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:12:42.995584 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:42.995589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:42.995804 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 16:12:42.995947 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 16:12:42.996172 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
I20260806 16:12:42.998142 27582 heartbeater.cc:347] Connected to a master server at 127.18.90.62:33657
I20260806 16:12:42.998387 27582 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:42.998848 27582 heartbeater.cc:511] Master 127.18.90.62:33657 requested a full tablet report, sending...
I20260806 16:12:42.999800 27117 ts_manager.cc:194] Registered new tserver with Master: c0507b14af5749dfbe9831472fa18835 (127.18.90.3:33117)
I20260806 16:12:43.000484 27117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:40979
I20260806 16:12:43.006377 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab.
Entry for principal oryx/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab.
WARNING: no policy specified for oryx/127.18.90.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab.
Entry for principal oryx/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab.
WARNING: no policy specified for oryx/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab.
Entry for principal oryx/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab.
I20260806 16:12:43.078743 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27179
I20260806 16:12:43.085227 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27315
I20260806 16:12:43.091053 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27451
I20260806 16:12:43.096590 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27085
I20260806 16:12:43.103369 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33657
--webserver_interface=127.18.90.62
--webserver_port=41151
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:12:43.213788 27591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:43.214028 27591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:43.214066 27591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:43.217916 27591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:43.217965 27591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:43.217983 27591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:43.217999 27591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:43.218014 27591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:43.223215 27591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:43.224115 27591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:43.225103 27591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:43.233508 27599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:43.233559 27597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.233619 27591 server_base.cc:1034] running on GCE node
W20260806 16:12:43.233522 27596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.234301 27591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:43.235348 27591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:43.236541 27591 hybrid_clock.cc:648] HybridClock initialized: now 1786032763236524 us; error 35 us; skew 500 ppm
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:43.239233 27591 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:12:43.240077 27591 webserver.cc:460] Webserver started at http://127.18.90.62:41151/ using document root <none> and password file <none>
I20260806 16:12:43.240691 27591 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:43.240813 27591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:43.244630 27591 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 16:12:43.246734 27606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.248016 27591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:43.248188 27591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "94b278c8ba724628903b5dbe202ddc64"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "c86ac85b9e8b254483c62b22586eea1f"
server_key_iv: "c760d43d65c9cebddb6a6f31ebd90103"
server_key_version: "encryptionkey@0"
I20260806 16:12:43.248803 27591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:43.261499 27591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:43.264104 27591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:43.264324 27591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:43.270989 27591 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:33657
I20260806 16:12:43.271003 27658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:33657 every 8 connection(s)
I20260806 16:12:43.271957 27591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:43.275851 27659 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:12:43.277534 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Bootstrap starting.
I20260806 16:12:43.279724 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27591
I20260806 16:12:43.280248 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:33661
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:33657
--webserver_port=35943
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:12:43.294782 27659 log.cc:824] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:43.301007 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:12:43.301365 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Bootstrap complete.
I20260806 16:12:43.305686 27659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:43.306092 27659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b278c8ba724628903b5dbe202ddc64, State: Initialized, Role: FOLLOWER
I20260806 16:12:43.306627 27659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:43.306773 27659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:43.306861 27659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:43.306979 27659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:12:43.308913 27659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:43.309229 27659 leader_election.cc:302] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b278c8ba724628903b5dbe202ddc64; no voters:
I20260806 16:12:43.309561 27659 leader_election.cc:288] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:12:43.309643 27663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:12:43.309892 27663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [term 2 LEADER]: Becoming Leader. State: Replica: 94b278c8ba724628903b5dbe202ddc64, State: Running, Role: LEADER
I20260806 16:12:43.310230 27663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } }
I20260806 16:12:43.310753 27659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:43.311621 27664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94b278c8ba724628903b5dbe202ddc64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } } }
I20260806 16:12:43.311756 27664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:43.312098 27665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b278c8ba724628903b5dbe202ddc64. Latest consensus state: current_term: 2 leader_uuid: "94b278c8ba724628903b5dbe202ddc64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b278c8ba724628903b5dbe202ddc64" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33657 } } }
I20260806 16:12:43.312203 27665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:43.312089 27673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:43.315414 27673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:43.316082 27673 catalog_manager.cc:1259] Loaded cluster ID: f2c162c9915b4454a69b79ca15b0c776
I20260806 16:12:43.316145 27673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:43.318462 27673 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:43.319103 27673 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94b278c8ba724628903b5dbe202ddc64: Loaded TSK: 0
I20260806 16:12:43.320293 27673 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:12:43.393246 27661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:43.393528 27661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:43.393606 27661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:43.397682 27661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:43.397760 27661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:43.397848 27661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:43.403487 27661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.1.keytab
--principal=oryx/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:33661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=35943
--tserver_master_addrs=127.18.90.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:43.404583 27661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:43.405576 27661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:43.416340 27687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.416535 27661 server_base.cc:1034] running on GCE node
W20260806 16:12:43.416358 27688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:43.416543 27690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.417148 27661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:43.417827 27661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:43.419009 27661 hybrid_clock.cc:648] HybridClock initialized: now 1786032763418988 us; error 32 us; skew 500 ppm
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:43.422006 27661 init.cc:377] Logged in from keytab as oryx/127.18.90.1@KRBTEST.COM (short username oryx)
I20260806 16:12:43.422876 27661 webserver.cc:460] Webserver started at http://127.18.90.1:35943/ using document root <none> and password file <none>
I20260806 16:12:43.423452 27661 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:43.423580 27661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:43.427712 27661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:12:43.429945 27697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.431295 27661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:43.431442 27661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "0ab3e99ca28b42efb2e7af5c1dca71da"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "06ec6593778c08699edac6e4096e6417"
server_key_iv: "7e21e38921f9013ed816059e9f878713"
server_key_version: "encryptionkey@0"
I20260806 16:12:43.432027 27661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:43.449551 27661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:43.453054 27661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:43.453284 27661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:43.453925 27661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:43.454811 27661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:43.454893 27661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.454990 27661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:43.455041 27661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.464737 27661 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:33661
I20260806 16:12:43.464759 27811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:33661 every 8 connection(s)
I20260806 16:12:43.465741 27661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:43.467386 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27661
I20260806 16:12:43.467876 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:44201
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:33657
--webserver_port=38145
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:12:43.480805 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.467805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:48931 (local address 127.18.90.62:33657)
0806 16:12:43.468436 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.468456 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.468716 (+ 260us) server_negotiation.cc:407] Connection header received
0806 16:12:43.470222 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.470233 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.470518 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.470835 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.472369 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.473429 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.475292 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.475745 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.475862 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.480045 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:48931: BlockingRecv error: recv got EOF from 127.18.90.1:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":144,"threads_started":1}
I20260806 16:12:43.480911 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.468064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:48931)
0806 16:12:43.468566 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.468606 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.469812 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.470920 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.470928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.471471 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.472145 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.472168 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.473583 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.473590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.475161 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.475177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.475915 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.475921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.476147 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.476850 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.476874 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.479850 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":167,"threads_started":1}
W20260806 16:12:43.481397 27812 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:33657 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.90.62:33657: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:43.487099 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.481812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:44169)
0806 16:12:43.481997 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.482011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.482082 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.482294 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.482296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.482422 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.482515 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.482519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.483513 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.483515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.484024 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.484032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.484579 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.484582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.484646 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.485156 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.485168 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.487007 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119}
W20260806 16:12:43.487578 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.481929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:44169 (local address 127.18.90.62:33657)
0806 16:12:43.482058 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.482062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.482077 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:43.482146 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.482149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.482191 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.482269 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.482615 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.483405 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.484126 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.484481 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.484577 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.487190 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:44169: BlockingRecv error: recv got EOF from 127.18.90.1:44169 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:43.492816 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.487672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:53075)
0806 16:12:43.487842 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.487859 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.487946 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.488180 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.488182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.488357 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.488430 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.488435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.489418 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.489420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.489923 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.489931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.490485 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.490487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.490541 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.491017 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.491028 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.492734 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260806 16:12:43.493016 27812 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:12:43.493208 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.487724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:53075 (local address 127.18.90.62:33657)
0806 16:12:43.487840 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.487844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.487922 (+ 78us) server_negotiation.cc:407] Connection header received
0806 16:12:43.488031 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.488033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.488076 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.488153 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.488533 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.489296 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.490030 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.490381 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.490479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.492899 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:53075: BlockingRecv error: recv got EOF from 127.18.90.1:53075 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 16:12:43.585568 27816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:43.585834 27816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:43.585881 27816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:43.589785 27816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:43.589838 27816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:43.589900 27816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:43.595010 27816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.2.keytab
--principal=oryx/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=38145
--tserver_master_addrs=127.18.90.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:43.595970 27816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:43.596944 27816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:43.606254 27822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.606298 27816 server_base.cc:1034] running on GCE node
W20260806 16:12:43.606400 27821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:43.606267 27824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.606896 27816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:43.607522 27816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:43.608702 27816 hybrid_clock.cc:648] HybridClock initialized: now 1786032763608684 us; error 28 us; skew 500 ppm
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:43.611495 27816 init.cc:377] Logged in from keytab as oryx/127.18.90.2@KRBTEST.COM (short username oryx)
I20260806 16:12:43.612479 27816 webserver.cc:460] Webserver started at http://127.18.90.2:38145/ using document root <none> and password file <none>
I20260806 16:12:43.613015 27816 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:43.613121 27816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:43.616899 27816 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 16:12:43.618816 27831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.619941 27816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:43.620069 27816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "ee616142b7b7466291838bcf93571169"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "76dd6cec45db5f972ac653d6e4038301"
server_key_iv: "c9eac7cefd7f7a85ed3c4bc339fe6fba"
server_key_version: "encryptionkey@0"
I20260806 16:12:43.620663 27816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:43.643906 27816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:43.647289 27816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:43.647508 27816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:43.648118 27816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:43.648983 27816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:43.649055 27816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.649129 27816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:43.649173 27816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.658422 27816 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44201
I20260806 16:12:43.658449 27945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44201 every 8 connection(s)
I20260806 16:12:43.659423 27816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:43.665995 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27816
I20260806 16:12:43.666464 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:33117
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:33657
--webserver_port=35321
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:12:43.670974 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.661240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:53969 (local address 127.18.90.62:33657)
0806 16:12:43.661370 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.661379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.662191 (+ 812us) server_negotiation.cc:407] Connection header received
0806 16:12:43.663509 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.663512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.663577 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.663653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.664876 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.665411 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.666697 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.666925 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.666986 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.670870 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:53969: BlockingRecv error: recv got EOF from 127.18.90.2:53969 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:12:43.671864 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.661601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:53969)
0806 16:12:43.662049 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.662090 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.663290 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.663821 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.663828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.664162 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.664721 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.664735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.665538 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.665541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.666533 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.666547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.667179 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.667181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.667321 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.667943 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.667959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.670757 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
W20260806 16:12:43.672377 27946 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:33657 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.90.62:33657: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:43.677865 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.672821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:42329)
0806 16:12:43.672929 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.672944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.673020 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.673281 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.673284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.673445 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.673527 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.673531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.674239 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.674245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.674781 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.674789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.675232 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.675235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.675329 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.675821 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.675833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.677724 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 16:12:43.677953 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.672933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:42329 (local address 127.18.90.62:33657)
0806 16:12:43.673087 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.673092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.673104 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:12:43.673145 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.673147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.673180 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.673264 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.673638 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.674133 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.674904 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.675123 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.675231 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.677832 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:42329: BlockingRecv error: recv got EOF from 127.18.90.2:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:43.683122 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.678412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:60253)
0806 16:12:43.678499 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.678515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.678588 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.678838 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.678841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.678991 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.679054 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.679058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.679746 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.679751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.680296 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.680306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.680750 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.680753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.680828 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.681303 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.681316 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.682998 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 16:12:43.683207 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.678509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:60253 (local address 127.18.90.62:33657)
0806 16:12:43.678648 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.678655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.678668 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:43.678706 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.678708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.678739 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.678823 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.679163 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.679640 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.680422 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.680634 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.680747 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.683096 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:60253: BlockingRecv error: recv got EOF from 127.18.90.2:60253 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 16:12:43.683364 27946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:12:43.779599 27949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:43.779840 27949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:43.779881 27949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:43.783731 27949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:43.783785 27949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:43.783880 27949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:43.788962 27949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/krb5kdc/oryx_127.18.90.3.keytab
--principal=oryx/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:33117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=35321
--tserver_master_addrs=127.18.90.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:43.789948 27949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:43.790940 27949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:43.800458 27954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:43.800500 27957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.800575 27949 server_base.cc:1034] running on GCE node
W20260806 16:12:43.800491 27955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:43.801152 27949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:43.801709 27949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:43.802898 27949 hybrid_clock.cc:648] HybridClock initialized: now 1786032763802872 us; error 48 us; skew 500 ppm
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:12:43.805545 27949 init.cc:377] Logged in from keytab as oryx/127.18.90.3@KRBTEST.COM (short username oryx)
I20260806 16:12:43.806418 27949 webserver.cc:460] Webserver started at http://127.18.90.3:35321/ using document root <none> and password file <none>
I20260806 16:12:43.806964 27949 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:43.807076 27949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:43.810818 27949 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:43.813052 27964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.814349 27949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:12:43.814477 27949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "c0507b14af5749dfbe9831472fa18835"
format_stamp: "Formatted at 2026-08-06 16:12:42 on dist-test-slave-0b58"
server_key: "3eb5576ab561cf58d747ed55eef17ee6"
server_key_iv: "b8ba9ca0fa12db505fe759f74915077f"
server_key_version: "encryptionkey@0"
I20260806 16:12:43.815012 27949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:43.831362 27949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:43.834156 27949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:43.834363 27949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:43.834931 27949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:43.835734 27949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:43.835801 27949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.835871 27949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:43.835913 27949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:43.845090 27949 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:33117
I20260806 16:12:43.845119 28078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:33117 every 8 connection(s)
I20260806 16:12:43.846077 27949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:43.854655 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 27949
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:12:43.857246 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.847816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:45257 (local address 127.18.90.62:33657)
0806 16:12:43.847948 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.847952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.848791 (+ 839us) server_negotiation.cc:407] Connection header received
0806 16:12:43.850084 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.850088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.850151 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.850231 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.851356 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.851908 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.853143 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.853366 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.853479 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.857122 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:45257: BlockingRecv error: recv got EOF from 127.18.90.3:45257 (error 108)
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":49}
I20260806 16:12:43.857765 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.848121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:45257)
0806 16:12:43.848635 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.848677 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.849870 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.850408 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.850414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.850751 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.851200 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.851213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.852065 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.852074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.853009 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.853019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.853535 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.853540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.853708 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.854315 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.854331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.856911 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":202,"threads_started":1}
W20260806 16:12:43.858228 28079 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:33657 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.90.62:33657: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:43.863790 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.858641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:52221)
0806 16:12:43.858728 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.858743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.858814 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.859049 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.859051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.859197 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.859284 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.859288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.860015 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.860020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.860609 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.860618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.861217 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.861221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.861323 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.861833 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.861852 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.863678 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 16:12:43.863917 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.858724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:52221 (local address 127.18.90.62:33657)
0806 16:12:43.858835 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.858841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.858856 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:43.858894 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.858897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.858932 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.859035 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.859426 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.859906 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.860722 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.861070 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.861174 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.863788 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:52221: BlockingRecv error: recv got EOF from 127.18.90.3:52221 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:12:43.864982 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.855740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.18.90.62:33657)
0806 16:12:43.855984 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.855989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.856008 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:12:43.856037 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.856039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.856072 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.856187 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.856892 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.857583 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.858804 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.859035 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.859136 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.860617 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:43.860622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:43.860635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:12:43.860659 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:43.862657 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:43.863155 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:43.863160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:43.863164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:43.863305 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:12:43.863577 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:12:43.863580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:12:43.863582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:12:43.864025 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:43.864218 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:43.864562 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 16:12:43.864818 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:43 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:43.869648 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:43.864419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:43305)
0806 16:12:43.864560 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:43.864574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:43.864636 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:43.864888 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:43.864890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:43.865004 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:43.865126 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.865134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.866028 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.866031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.866820 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:43.866831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.867314 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.867318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:43.867399 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.867861 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:43.867873 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:43.869567 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 16:12:43.869776 27814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:43.864462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:43305 (local address 127.18.90.62:33657)
0806 16:12:43.864593 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:43.864597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:43.864610 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:43.864736 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:43.864739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:43.864781 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:43.864853 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:43.865237 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.865910 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.866971 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:43.867204 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:43.867307 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:43.869675 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:43305: BlockingRecv error: recv got EOF from 127.18.90.3:43305 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 16:12:43.870038 28079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 16:12:44 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:44 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:44.499462 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:44.493674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:54239)
0806 16:12:44.494043 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:44.494058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:44.494159 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:44.494409 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:44.494411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:44.494596 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:44.494697 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:44.494702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.495476 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.495479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:44.496060 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:44.496069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.496579 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.496582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:44.496652 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:44.497160 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:44.497171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:44.499272 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
W20260806 16:12:44.499540 28085 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:44.493765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:54239 (local address 127.18.90.62:33657)
0806 16:12:44.494109 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:44.494113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:44.494126 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:44.494247 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:44.494249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:44.494294 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:44.494367 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:44.494817 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.495362 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.496172 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.496477 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.496558 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:44.499406 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:54239: BlockingRecv error: recv got EOF from 127.18.90.1:54239 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
Aug 06 16:12:44 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:44 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:44.689781 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:44.684061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:40353)
0806 16:12:44.684381 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:44.684395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:44.684500 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:44.684751 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:44.684753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:44.684933 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:44.685029 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:44.685034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.685810 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.685813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:44.686427 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:44.686436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.686882 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.686886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:44.686988 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:44.687566 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:44.687579 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:44.689638 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":72,"threads_started":1}
W20260806 16:12:44.689940 28085 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:44.684216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:40353 (local address 127.18.90.62:33657)
0806 16:12:44.684401 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:44.684405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:44.684449 (+ 44us) server_negotiation.cc:407] Connection header received
0806 16:12:44.684588 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:44.684591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:44.684636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:44.684707 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:44.685139 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.685665 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.686536 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.686757 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.686835 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:44.689784 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:40353: BlockingRecv error: recv got EOF from 127.18.90.2:40353 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 16:12:44 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:44 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:44.876094 28087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:44.870709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:53477)
0806 16:12:44.870985 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:44.870999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:44.871091 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:44.871363 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:44.871365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:44.871538 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:44.871631 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:44.871636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.872372 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.872375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:44.872922 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:44.872929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.873348 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.873351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:44.873409 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:44.873966 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:44.873978 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:44.875964 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
W20260806 16:12:44.876317 28085 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:44.870846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:53477 (local address 127.18.90.62:33657)
0806 16:12:44.871021 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:44.871025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:44.871041 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:12:44.871203 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:44.871205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:44.871254 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:44.871322 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:44.871730 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.872276 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.873023 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:44.873244 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:44.873320 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:44.876125 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:53477: BlockingRecv error: recv got EOF from 127.18.90.3:53477 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 16:12:45 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:45 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:45.505743 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:45.500299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:55023)
0806 16:12:45.500599 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:45.500618 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:45.500733 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:45.500965 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:45.500967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:45.501154 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:45.501257 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:45.501276 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.502017 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.502019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:45.502582 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:45.502589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.503013 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.503015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:45.503077 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:45.503680 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:45.503692 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:45.505596 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
W20260806 16:12:45.505950 28089 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:45.500445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:55023 (local address 127.18.90.62:33657)
0806 16:12:45.500742 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:45.500745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:45.500759 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:45.500811 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:45.500813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:45.500855 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:45.500923 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:45.501383 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.501921 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.502692 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.502911 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.502987 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:45.505764 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:55023: BlockingRecv error: recv got EOF from 127.18.90.1:55023 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
Aug 06 16:12:45 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:45 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:45.696419 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:45.690644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:55801)
0806 16:12:45.690976 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:45.690990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:45.691132 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:45.691419 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:45.691422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:45.691654 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:45.691777 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:45.691784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.692625 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.692630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:45.693205 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:45.693214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.693707 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.693711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:45.693805 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:45.694351 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:45.694364 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:45.696182 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
W20260806 16:12:45.696466 28089 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:45.690740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:55801 (local address 127.18.90.62:33657)
0806 16:12:45.690918 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:45.690922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:45.691067 (+ 145us) server_negotiation.cc:407] Connection header received
0806 16:12:45.691237 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:45.691241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:45.691298 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:45.691390 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:45.691907 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.692510 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.693350 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.693584 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.693699 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:45.696354 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:55801: BlockingRecv error: recv got EOF from 127.18.90.2:55801 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 16:12:45 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:45 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:45.882529 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:45.876966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:43025)
0806 16:12:45.877280 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:45.877295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:45.877386 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:45.877645 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:45.877647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:45.877836 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:45.877933 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:45.877937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.878747 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.878750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:45.879368 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:45.879377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.879801 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.879804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:45.879868 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:45.880395 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:45.880407 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:45.882329 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
W20260806 16:12:45.882572 28089 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:45.877176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:43025 (local address 127.18.90.62:33657)
0806 16:12:45.877346 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:45.877350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:45.877363 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:45.877476 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:45.877478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:45.877527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:45.877594 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:45.878077 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.878640 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.879478 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:45.879700 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:45.879776 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:45.882466 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:43025: BlockingRecv error: recv got EOF from 127.18.90.3:43025 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 16:12:46 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:46 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:46.515609 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:46.506559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:43745)
0806 16:12:46.506849 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:46.506863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:46.506953 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:46.507277 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:46.507280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:46.507496 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:46.507598 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:46.507604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.508564 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.508568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:46.509127 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:46.509136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.509617 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.509621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:46.509708 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:46.513529 (+ 3821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:46.513542 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:46.515449 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
W20260806 16:12:46.515780 28093 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:46.506639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:43745 (local address 127.18.90.62:33657)
0806 16:12:46.506925 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:46.506928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:46.506942 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:46.507052 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:46.507056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:46.507110 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:46.507216 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:46.507752 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.508466 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.509248 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.509494 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.509611 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:46.515611 (+ 6000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:43745: BlockingRecv error: recv got EOF from 127.18.90.1:43745 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":127,"threads_started":1}
Aug 06 16:12:46 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:46 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:46.703397 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:46.697430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:40457)
0806 16:12:46.697737 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:46.697754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:46.697885 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:46.698201 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:46.698204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:46.698438 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:46.698535 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:46.698541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.699337 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.699342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:46.699940 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:46.699949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.700538 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.700542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:46.700640 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:46.701173 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:46.701185 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:46.703186 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
W20260806 16:12:46.703452 28093 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:46.697545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:40457 (local address 127.18.90.62:33657)
0806 16:12:46.697741 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:46.697748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:46.697816 (+ 68us) server_negotiation.cc:407] Connection header received
0806 16:12:46.697998 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:46.698001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:46.698056 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:46.698182 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:46.698660 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.699215 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.700078 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.700408 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.700532 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:46.703337 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:40457: BlockingRecv error: recv got EOF from 127.18.90.2:40457 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 16:12:46 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:46 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:46.889416 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:46.883480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:56069)
0806 16:12:46.883763 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:46.883777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:46.883883 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:46.884134 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:46.884136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:46.884356 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:46.884452 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:46.884458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.885244 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.885247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:46.886082 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:46.886094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.886648 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.886652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:46.886738 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:46.887266 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:46.887288 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:46.889267 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
W20260806 16:12:46.889540 28093 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:46.883557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:56069 (local address 127.18.90.62:33657)
0806 16:12:46.883739 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:46.883744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:46.883853 (+ 109us) server_negotiation.cc:407] Connection header received
0806 16:12:46.883965 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:46.883967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:46.884023 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:46.884093 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:46.884557 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.885107 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.886232 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:46.886520 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:46.886633 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:46.889417 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:56069: BlockingRecv error: recv got EOF from 127.18.90.3:56069 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 16:12:47 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:47 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:47.522512 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:47.516560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:33533)
0806 16:12:47.516910 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:47.516925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:47.517032 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:47.517336 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:47.517339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:47.517549 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:47.517654 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:47.517661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.518490 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.518497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:47.519180 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:47.519191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.519766 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.519770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:47.519851 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:47.520430 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:47.520442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:47.522391 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":102,"threads_started":1}
W20260806 16:12:47.522670 28097 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:47.516642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:33533 (local address 127.18.90.62:33657)
0806 16:12:47.516929 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:47.516933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:47.516992 (+ 59us) server_negotiation.cc:407] Connection header received
0806 16:12:47.517138 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:47.517140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:47.517194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:47.517273 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:47.517794 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.518347 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.519313 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.519551 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.519702 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:47.522530 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:33533: BlockingRecv error: recv got EOF from 127.18.90.1:33533 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
Aug 06 16:12:47 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:47 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:47.710260 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:47.704421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:34829)
0806 16:12:47.704709 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:47.704723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:47.704825 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:47.705112 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:47.705115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:47.705335 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:47.705438 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:47.705443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.706454 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.706459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:47.707033 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:47.707042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.707530 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.707534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:47.707625 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:47.708226 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:47.708238 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:47.710119 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
W20260806 16:12:47.710420 28097 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:47.704572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:34829 (local address 127.18.90.62:33657)
0806 16:12:47.704755 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:47.704759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:47.704774 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:47.704919 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:47.704922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:47.704967 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:47.705038 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:47.705589 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.706336 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.707164 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.707408 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.707523 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:47.710263 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:34829: BlockingRecv error: recv got EOF from 127.18.90.2:34829 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 16:12:47 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:47 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:47.895983 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:47.890451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:56325)
0806 16:12:47.890804 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:47.890824 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:47.890983 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:47.891272 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:47.891275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:47.891490 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:47.891580 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:47.891585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.892385 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.892391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:47.892953 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:47.892961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.893455 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.893459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:47.893558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:47.894026 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:47.894036 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:47.895815 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
W20260806 16:12:47.896034 28097 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:47.890623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:56325 (local address 127.18.90.62:33657)
0806 16:12:47.890805 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:47.890811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:47.890916 (+ 105us) server_negotiation.cc:407] Connection header received
0806 16:12:47.891091 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:47.891095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:47.891145 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:47.891258 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:47.891706 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.892237 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.893079 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:47.893334 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:47.893448 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:47.895931 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:56325: BlockingRecv error: recv got EOF from 127.18.90.3:56325 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 16:12:48 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:48 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:48.529433 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:48.523524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:37389)
0806 16:12:48.523889 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:48.523904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:48.523993 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:48.524274 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:48.524276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:48.524499 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:48.524640 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:48.524648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.525497 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.525500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:48.526297 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:48.526309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.526776 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.526779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:48.526842 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:48.527432 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:48.527444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:48.529306 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":82,"threads_started":1}
W20260806 16:12:48.529608 28101 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:48.523592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:37389 (local address 127.18.90.62:33657)
0806 16:12:48.523934 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:48.523937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:48.523952 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:48.524086 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:48.524088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:48.524133 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:48.524210 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:48.524766 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.525386 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.526444 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.526671 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.526757 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:48.529460 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:37389: BlockingRecv error: recv got EOF from 127.18.90.1:37389 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
Aug 06 16:12:48 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:48 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:48.717020 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:48.711057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:43367)
0806 16:12:48.711460 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:48.711477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:48.711606 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:48.711898 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:48.711901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:48.712116 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:48.712211 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:48.712217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.713136 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.713140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:48.713736 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:48.713745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.714218 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.714221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:48.714312 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:48.714888 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:48.714901 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:48.716842 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
W20260806 16:12:48.717080 28101 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:48.711181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:43367 (local address 127.18.90.62:33657)
0806 16:12:48.711329 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:48.711336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:48.711553 (+ 217us) server_negotiation.cc:407] Connection header received
0806 16:12:48.711726 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:48.711729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:48.711777 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:48.711866 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:48.712424 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.713023 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.713871 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.714099 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.714209 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:48.716970 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:43367: BlockingRecv error: recv got EOF from 127.18.90.2:43367 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 16:12:48 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:48 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:48.902570 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:48.896870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:55013)
0806 16:12:48.897176 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:48.897198 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:48.897328 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:48.897596 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:48.897598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:48.897838 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:48.897930 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:48.897936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.898711 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.898716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:48.899306 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:48.899314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.899797 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.899800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:48.899886 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:48.900487 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:48.900508 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:48.902394 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
W20260806 16:12:48.902637 28101 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:48.896967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:55013 (local address 127.18.90.62:33657)
0806 16:12:48.897104 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:48.897109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:48.897292 (+ 183us) server_negotiation.cc:407] Connection header received
0806 16:12:48.897414 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:48.897419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:48.897474 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:48.897565 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:48.898058 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.898599 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.899445 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:48.899670 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:48.899784 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:48.902529 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:55013: BlockingRecv error: recv got EOF from 127.18.90.3:55013 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 16:12:49 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:49 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:49.536006 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:49.530388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:50793)
0806 16:12:49.530729 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:49.530747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:49.530869 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:49.531209 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:49.531215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:49.531466 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:49.531585 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:49.531600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.532356 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.532358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:49.532908 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:49.532915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.533322 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.533324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:49.533386 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:49.533883 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:49.533895 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:49.535821 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":85,"threads_started":1}
W20260806 16:12:49.536175 28105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:49.530442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:50793 (local address 127.18.90.62:33657)
0806 16:12:49.530719 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:49.530723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:49.530827 (+ 104us) server_negotiation.cc:407] Connection header received
0806 16:12:49.530981 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:49.530984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:49.531047 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:49.531138 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:49.531703 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.532241 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.533004 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.533226 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.533314 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:49.535974 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:50793: BlockingRecv error: recv got EOF from 127.18.90.1:50793 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
Aug 06 16:12:49 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:49 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:49.723974 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:49.718106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:45139)
0806 16:12:49.718426 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:49.718441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:49.718548 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:49.718808 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:49.718810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:49.719077 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:49.719180 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:49.719186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.719950 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.719952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:49.720717 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:49.720730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.721246 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.721250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:49.721329 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:49.721933 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:49.721946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:49.723851 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
W20260806 16:12:49.724159 28105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:49.718184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:45139 (local address 127.18.90.62:33657)
0806 16:12:49.718342 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:49.718346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:49.718513 (+ 167us) server_negotiation.cc:407] Connection header received
0806 16:12:49.718633 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:49.718636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:49.718684 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:49.718776 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:49.719300 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.719841 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.720864 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.721105 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.721215 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:49.723994 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:45139: BlockingRecv error: recv got EOF from 127.18.90.2:45139 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 16:12:49 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:49 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:49.908522 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:49.903428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:54629)
0806 16:12:49.903684 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:49.903698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:49.903807 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:49.904023 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:49.904025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:49.904208 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:49.904385 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:49.904392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.905096 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.905098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:49.905644 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:49.905652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.906059 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.906061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:49.906122 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:49.906622 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:49.906633 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:49.908402 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
W20260806 16:12:49.908665 28105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:49.903526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:54629 (local address 127.18.90.62:33657)
0806 16:12:49.903696 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:49.903699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:49.903748 (+ 49us) server_negotiation.cc:407] Connection header received
0806 16:12:49.903898 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:49.903900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:49.903946 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:49.904010 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:49.904487 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.905005 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.905742 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:49.905960 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:49.906033 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:49.908532 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:54629: BlockingRecv error: recv got EOF from 127.18.90.3:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 16:12:50 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:50 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:50.542829 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:50.537015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:55019)
0806 16:12:50.537386 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:50.537404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:50.537530 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:50.537884 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:50.537887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:50.538118 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:50.538258 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:50.538268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.539128 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.539131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:50.539689 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:50.539696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.540119 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.540122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:50.540185 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:50.540803 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:50.540817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:50.542699 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":96,"threads_started":1}
W20260806 16:12:50.543009 28109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:50.537079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:55019 (local address 127.18.90.62:33657)
0806 16:12:50.537438 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:50.537441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:50.537459 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:12:50.537637 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:50.537642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:50.537719 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:50.537819 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:50.538391 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.539016 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.539796 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.540024 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.540098 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:50.542834 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:55019: BlockingRecv error: recv got EOF from 127.18.90.1:55019 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":170,"threads_started":1}
Aug 06 16:12:50 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:50 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:50.730738 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:50.724887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:52149)
0806 16:12:50.725229 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:50.725245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:50.725372 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:50.725625 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:50.725628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:50.725834 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:50.725934 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:50.725941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.726815 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.726820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:50.727561 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:50.727572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.728103 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.728106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:50.728174 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:50.728819 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:50.728832 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:50.730616 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
W20260806 16:12:50.731007 28109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:50.724959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:52149 (local address 127.18.90.62:33657)
0806 16:12:50.725129 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:50.725134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:50.725337 (+ 203us) server_negotiation.cc:407] Connection header received
0806 16:12:50.725460 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:50.725462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:50.725512 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:50.725582 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:50.726042 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.726697 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.727678 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.727995 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.728077 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:50.730757 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:52149: BlockingRecv error: recv got EOF from 127.18.90.2:52149 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 16:12:50 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:50 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:50.915318 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:50.909442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:53961)
0806 16:12:50.909859 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:50.909873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:50.910018 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:50.910311 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:50.910314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:50.910532 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:50.910623 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:50.910628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.911447 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.911453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:50.912088 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:50.912097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.912621 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.912625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:50.912726 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:50.913253 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:50.913264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:50.915146 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":97,"threads_started":1}
W20260806 16:12:50.915385 28109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:50.909516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:53961 (local address 127.18.90.62:33657)
0806 16:12:50.909753 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:50.909761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:50.909953 (+ 192us) server_negotiation.cc:407] Connection header received
0806 16:12:50.910132 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:50.910135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:50.910199 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:50.910286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:50.910778 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.911333 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.912208 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:50.912495 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:50.912605 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:50.915275 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:53961: BlockingRecv error: recv got EOF from 127.18.90.3:53961 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 06 16:12:51 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:51 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:51.553798 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:51.543670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:59841)
0806 16:12:51.544030 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:51.544045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:51.544167 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:51.544488 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:51.544491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:51.544744 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:51.544874 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:51.544881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.545699 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.545707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:51.546288 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:51.546296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.546708 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.546710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:51.546772 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:51.551528 (+ 4756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:51.551542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:51.553659 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
W20260806 16:12:51.554011 28113 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:51.543889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:59841 (local address 127.18.90.62:33657)
0806 16:12:51.544157 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:51.544165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:51.544179 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:51.544248 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:51.544251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:51.544333 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:51.544424 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:51.545015 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.545593 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.546388 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.546604 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.546677 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:51.553825 (+ 7148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:59841: BlockingRecv error: recv got EOF from 127.18.90.1:59841 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
Aug 06 16:12:51 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:51 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:51.737319 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:51.731545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:35461)
0806 16:12:51.731830 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:51.731845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:51.731935 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:51.732203 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:51.732205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:51.732447 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:51.732552 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:51.732557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.733450 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.733453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:51.734011 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:51.734019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.734466 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.734469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:51.734532 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:51.735024 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:51.735035 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:51.737198 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
W20260806 16:12:51.737468 28113 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:51.731609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:35461 (local address 127.18.90.62:33657)
0806 16:12:51.731882 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:51.731885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:51.731898 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:12:51.732022 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:51.732024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:51.732071 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:51.732163 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:51.732665 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.733349 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.734117 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.734360 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.734442 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:51.737345 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:35461: BlockingRecv error: recv got EOF from 127.18.90.2:35461 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Aug 06 16:12:51 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:51 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:51.922487 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:51.916217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:53483)
0806 16:12:51.916541 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:51.916556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:51.916663 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:51.916906 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:51.916909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:51.917096 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:51.917194 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:51.917199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.918211 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.918215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:51.919101 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:51.919113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.919647 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.919650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:51.919733 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:51.920332 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:51.920353 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:51.922310 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":123,"threads_started":1}
W20260806 16:12:51.922549 28113 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:51.916358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:53483 (local address 127.18.90.62:33657)
0806 16:12:51.916534 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:51.916537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:51.916626 (+ 89us) server_negotiation.cc:407] Connection header received
0806 16:12:51.916733 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:51.916735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:51.916782 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:51.916851 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:51.917338 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.918046 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.919238 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:51.919531 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:51.919641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:51.922440 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:53483: BlockingRecv error: recv got EOF from 127.18.90.3:53483 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 16:12:52 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:52 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:52.560631 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:52.554614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:49939)
0806 16:12:52.554949 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:52.554964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:52.555070 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:52.555328 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:52.555330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:52.555512 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:52.555614 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:52.555618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.556415 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.556417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:52.556972 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:52.556980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.557433 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.557436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:52.557497 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:52.558197 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:52.558210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:52.560511 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
W20260806 16:12:52.560818 28117 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:52.554728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:49939 (local address 127.18.90.62:33657)
0806 16:12:52.554973 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:52.554977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:52.555055 (+ 78us) server_negotiation.cc:407] Connection header received
0806 16:12:52.555151 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:52.555153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:52.555205 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:52.555284 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:52.555724 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.556331 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.557089 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.557329 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.557412 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:52.560642 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:49939: BlockingRecv error: recv got EOF from 127.18.90.1:49939 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
Aug 06 16:12:52 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:52 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:52.743764 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:52.738363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:43955)
0806 16:12:52.738631 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:52.738651 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:52.738752 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:52.739021 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:52.739024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:52.739235 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:52.739337 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:52.739343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.740125 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.740128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:52.740758 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:52.740768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.741186 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.741189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:52.741252 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:52.741752 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:52.741763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:52.743646 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
W20260806 16:12:52.743920 28117 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:52.738417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:43955 (local address 127.18.90.62:33657)
0806 16:12:52.738579 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:52.738583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:52.738723 (+ 140us) server_negotiation.cc:407] Connection header received
0806 16:12:52.738830 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:52.738832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:52.738892 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:52.738960 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:52.739475 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.740012 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.740869 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.741086 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.741164 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:52.743794 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:43955: BlockingRecv error: recv got EOF from 127.18.90.2:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 16:12:52 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:52 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:52.929019 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:52.923465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:41987)
0806 16:12:52.923775 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:52.923788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:52.923882 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:52.924100 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:52.924102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:52.924331 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:52.924429 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:52.924434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.925342 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.925345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:52.925900 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:52.925908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.926325 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.926327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:52.926389 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:52.926891 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:52.926903 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:52.928901 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":114,"threads_started":1}
W20260806 16:12:52.929155 28117 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:52.923517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:41987 (local address 127.18.90.62:33657)
0806 16:12:52.923652 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:52.923659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:52.923877 (+ 218us) server_negotiation.cc:407] Connection header received
0806 16:12:52.923960 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:52.923963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:52.924008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:52.924075 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:52.924553 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.925244 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.926002 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:52.926226 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:52.926301 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:52.929035 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:41987: BlockingRecv error: recv got EOF from 127.18.90.3:41987 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 16:12:53 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:53 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:53.567054 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:53.561453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:55675)
0806 16:12:53.561748 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:53.561762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:53.561857 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:53.562104 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:53.562106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:53.562313 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:53.562411 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:53.562416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.563250 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.563254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:53.563882 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:53.563894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.564374 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.564377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:53.564441 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:53.564953 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:53.564965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:53.566908 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
W20260806 16:12:53.567246 28121 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:53.561502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:55675 (local address 127.18.90.62:33657)
0806 16:12:53.561749 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:53.561753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:53.561838 (+ 85us) server_negotiation.cc:407] Connection header received
0806 16:12:53.561942 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:53.561944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:53.561989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:53.562067 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:53.562530 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.563087 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.564000 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.564248 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.564358 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:53.567062 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:55675: BlockingRecv error: recv got EOF from 127.18.90.1:55675 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":60,"threads_started":1}
Aug 06 16:12:53 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:53 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:53.750159 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:53.744737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:44123)
0806 16:12:53.745028 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:53.745042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:53.745138 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:53.745367 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:53.745369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:53.745545 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:53.745643 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:53.745648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.746378 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.746380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:53.746935 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:53.746942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.747368 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.747371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:53.747432 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:53.747925 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:53.747936 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:53.750008 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":103,"threads_started":1}
W20260806 16:12:53.750268 28121 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:53.744836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:44123 (local address 127.18.90.62:33657)
0806 16:12:53.745017 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:53.745021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:53.745095 (+ 74us) server_negotiation.cc:407] Connection header received
0806 16:12:53.745208 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:53.745210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:53.745252 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:53.745326 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:53.745751 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.746287 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.747043 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.747267 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.747348 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:53.750135 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:44123: BlockingRecv error: recv got EOF from 127.18.90.2:44123 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 16:12:53 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:53 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:53.935039 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:53.929868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:36609)
0806 16:12:53.930163 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:53.930177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:53.930299 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:53.930537 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:53.930539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:53.930726 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:53.930821 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:53.930826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.931546 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.931548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:53.932079 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:53.932087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.932567 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.932570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:53.932634 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:53.933117 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:53.933128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:53.934898 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":119,"threads_started":1}
W20260806 16:12:53.935185 28121 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:53.930014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:36609 (local address 127.18.90.62:33657)
0806 16:12:53.930163 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:53.930167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:53.930264 (+ 97us) server_negotiation.cc:407] Connection header received
0806 16:12:53.930380 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:53.930382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:53.930428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:53.930496 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:53.930922 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.931455 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.932181 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:53.932470 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:53.932548 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:53.935035 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:36609: BlockingRecv error: recv got EOF from 127.18.90.3:36609 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 16:12:54 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:54 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:54.574544 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:54.567916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:48769)
0806 16:12:54.568349 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:54.568366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:54.568491 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:54.568815 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:54.568818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:54.569034 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:54.569136 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:54.569142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.569966 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.569969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:54.570549 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:54.570559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.571053 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.571057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:54.571145 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:54.571811 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:54.571832 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:54.574359 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":84,"threads_started":1}
W20260806 16:12:54.574630 28125 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:54.568053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:48769 (local address 127.18.90.62:33657)
0806 16:12:54.568350 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:54.568356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:54.568427 (+ 71us) server_negotiation.cc:407] Connection header received
0806 16:12:54.568626 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:54.568628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:54.568685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:54.568784 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:54.569282 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.569844 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.570701 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.570935 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.571045 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:54.574502 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:48769: BlockingRecv error: recv got EOF from 127.18.90.1:48769 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":60,"threads_started":1}
Aug 06 16:12:54 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:54 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:54.756717 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:54.750969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:59517)
0806 16:12:54.751250 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:54.751275 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:54.751403 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:54.751721 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:54.751725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:54.751969 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:54.752127 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:54.752139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.753062 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.753065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:54.753694 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:54.753703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.754141 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.754144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:54.754211 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:54.754715 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:54.754726 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:54.756586 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":116,"threads_started":1}
W20260806 16:12:54.756848 28125 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:54.751079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:59517 (local address 127.18.90.62:33657)
0806 16:12:54.751253 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:54.751258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:54.751354 (+ 96us) server_negotiation.cc:407] Connection header received
0806 16:12:54.751518 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:54.751521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:54.751578 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:54.751681 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:54.752341 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.752956 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.753798 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.754025 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.754104 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:54.756722 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:59517: BlockingRecv error: recv got EOF from 127.18.90.2:59517 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 16:12:54 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:54 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:54.941601 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:54.935998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:55545)
0806 16:12:54.936302 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:54.936324 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:54.936427 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:54.936658 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:54.936660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:54.936847 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:54.936942 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:54.936947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.937752 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.937755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:54.938298 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:54.938305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.938708 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.938710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:54.938771 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:54.939333 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:54.939345 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:54.941480 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
W20260806 16:12:54.941816 28125 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:54.936069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:55545 (local address 127.18.90.62:33657)
0806 16:12:54.936218 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:54.936222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:54.936400 (+ 178us) server_negotiation.cc:407] Connection header received
0806 16:12:54.936497 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:54.936499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:54.936548 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:54.936621 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:54.937056 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.937642 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.938402 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:54.938610 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:54.938684 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:54.941675 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:55545: BlockingRecv error: recv got EOF from 127.18.90.3:55545 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 16:12:55 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:55 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:55.580773 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:55.575381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:38961)
0806 16:12:55.575678 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:55.575693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:55.575783 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:55.576039 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:55.576041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:55.576229 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:55.576356 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:55.576362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.577085 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.577087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:55.577643 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:55.577659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.578094 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.578097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:55.578164 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:55.578744 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:55.578757 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:55.580647 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
W20260806 16:12:55.580971 28129 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:55.575489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:38961 (local address 127.18.90.62:33657)
0806 16:12:55.575732 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:55.575735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:55.575749 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:55.575873 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:55.575876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:55.575920 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:55.576001 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:55.576458 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.576987 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.577765 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.577991 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.578072 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:55.580793 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:38961: BlockingRecv error: recv got EOF from 127.18.90.1:38961 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
Aug 06 16:12:55 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:55 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:55.762907 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:55.757626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:39917)
0806 16:12:55.757932 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:55.757952 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:55.758087 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:55.758401 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:55.758404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:55.758588 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:55.758692 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:55.758698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.759410 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.759413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:55.759960 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:55.759968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.760421 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.760423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:55.760490 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:55.760964 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:55.760975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:55.762789 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
W20260806 16:12:55.763074 28129 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:55.757698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:39917 (local address 127.18.90.62:33657)
0806 16:12:55.757881 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:55.757886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:55.758056 (+ 170us) server_negotiation.cc:407] Connection header received
0806 16:12:55.758221 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:55.758224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:55.758286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:55.758380 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:55.758793 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.759318 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.760057 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.760322 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.760399 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:55.762918 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:39917: BlockingRecv error: recv got EOF from 127.18.90.2:39917 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 16:12:55 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:55 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:55.948419 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:55.942534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:50381)
0806 16:12:55.942908 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:55.942926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:55.943053 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:55.943406 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:55.943412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:55.943678 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:55.943814 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:55.943823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.944720 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.944724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:55.945301 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:55.945308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.945789 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.945793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:55.945882 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:55.946408 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:55.946420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:55.948218 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":90,"threads_started":1}
W20260806 16:12:55.948513 28129 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:55.942648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:50381 (local address 127.18.90.62:33657)
0806 16:12:55.942833 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:55.942838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:55.943002 (+ 164us) server_negotiation.cc:407] Connection header received
0806 16:12:55.943159 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:55.943164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:55.943224 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:55.943335 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:55.943930 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.944604 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.945426 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:55.945663 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:55.945776 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:55.948388 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:50381: BlockingRecv error: recv got EOF from 127.18.90.3:50381 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 16:12:56 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:56 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:56.591405 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:56.581660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:49005)
0806 16:12:56.581978 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:56.581994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:56.582093 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:56.582340 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:56.582342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:56.582532 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:56.582631 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:56.582635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.583504 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.583508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:56.584489 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:56.584501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.584984 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.584988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:56.585073 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:56.589197 (+ 4124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:56.589211 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:56.591233 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
W20260806 16:12:56.591531 28133 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:56.581819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:49005 (local address 127.18.90.62:33657)
0806 16:12:56.582078 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:56.582081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:56.582096 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:12:56.582182 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:56.582184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:56.582228 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:56.582296 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:56.582739 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.583348 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.584614 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.584859 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.584975 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:56.591393 (+ 6418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:49005: BlockingRecv error: recv got EOF from 127.18.90.1:49005 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
Aug 06 16:12:56 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:56 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:56.769191 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:56.763745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:47227)
0806 16:12:56.764031 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:56.764045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:56.764144 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:56.764422 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:56.764424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:56.764609 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:56.764720 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:56.764726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.765450 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.765453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:56.766021 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:56.766029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.766487 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.766490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:56.766553 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:56.767042 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:56.767053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:56.769001 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
W20260806 16:12:56.769251 28133 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:56.763882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:47227 (local address 127.18.90.62:33657)
0806 16:12:56.764050 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:56.764054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:56.764098 (+ 44us) server_negotiation.cc:407] Connection header received
0806 16:12:56.764237 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:56.764239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:56.764298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:56.764376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:56.764817 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.765349 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.766123 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.766364 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.766444 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:56.769126 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:47227: BlockingRecv error: recv got EOF from 127.18.90.2:47227 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 16:12:56 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:56 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:56.955178 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:56.949322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:33005)
0806 16:12:56.949633 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:56.949647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:56.949793 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:56.950085 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:56.950088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:56.950311 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:56.950405 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:56.950411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.951262 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.951268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:56.951878 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:56.951887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.952429 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.952433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:56.952522 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:56.953198 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:56.953210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:56.955059 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
W20260806 16:12:56.955474 28133 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:56.949401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:33005 (local address 127.18.90.62:33657)
0806 16:12:56.949553 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:56.949558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:56.949739 (+ 181us) server_negotiation.cc:407] Connection header received
0806 16:12:56.949922 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:56.949924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:56.949972 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:56.950060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:56.950531 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.951127 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.952002 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:56.952295 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:56.952426 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:56.955213 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:33005: BlockingRecv error: recv got EOF from 127.18.90.3:33005 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 16:12:57 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:57 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:57.597922 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:57.592354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:41285)
0806 16:12:57.592638 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:57.592659 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:57.592774 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:57.593063 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:57.593065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:57.593259 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:57.593359 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:57.593364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.594117 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.594120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:57.594713 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:57.594722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.595137 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.595140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:57.595214 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:57.595783 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:57.595796 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:57.597729 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
W20260806 16:12:57.597985 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:57.592443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:41285 (local address 127.18.90.62:33657)
0806 16:12:57.592779 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:57.592784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:57.592798 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:12:57.592871 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:57.592876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:57.592929 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:57.593017 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:57.593463 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.594016 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.594817 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.595032 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.595110 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:57.597858 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:41285: BlockingRecv error: recv got EOF from 127.18.90.1:41285 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":69,"threads_started":1}
Aug 06 16:12:57 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:57 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:57.775624 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:57.770160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:47107)
0806 16:12:57.770457 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:57.770471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:57.770579 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:57.770818 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:57.770820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:57.771006 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:57.771106 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:57.771111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.771839 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.771841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:57.772529 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:57.772543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.772973 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.772975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:57.773037 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:57.773615 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:57.773628 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:57.775498 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
W20260806 16:12:57.775748 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:57.770237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:47107 (local address 127.18.90.62:33657)
0806 16:12:57.770394 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:57.770400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:57.770564 (+ 164us) server_negotiation.cc:407] Connection header received
0806 16:12:57.770661 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:57.770663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:57.770709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:57.770783 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:57.771207 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.771720 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.772644 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.772872 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.772954 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:57.775617 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:47107: BlockingRecv error: recv got EOF from 127.18.90.2:47107 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 16:12:57 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:57 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.3@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:57.961577 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:57.955974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.3:51839)
0806 16:12:57.956370 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:57.956384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:57.956487 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:57.956710 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:57.956712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:57.956899 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:57.956995 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:57.957000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.957711 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.957713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:57.958279 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:57.958286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.958691 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.958693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:57.958757 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:57.959349 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:57.959361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:57.961444 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":173,"threads_started":1}
W20260806 16:12:57.961730 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:57.956077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:51839 (local address 127.18.90.62:33657)
0806 16:12:57.956220 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:57.956224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:57.956452 (+ 228us) server_negotiation.cc:407] Connection header received
0806 16:12:57.956553 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:57.956555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:57.956601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:57.956669 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:57.957093 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.957614 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.958376 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:57.958587 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:57.958660 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:57.961593 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.3:51839: BlockingRecv error: recv got EOF from 127.18.90.3:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 16:12:58 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:58 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.1@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:58.604439 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:58.598884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.1:48511)
0806 16:12:58.599192 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:58.599215 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:58.599316 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:58.599535 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:58.599537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:58.599716 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:58.599813 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:58.599817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.600722 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.600725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:58.601298 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:58.601306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.601718 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.601720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:58.601784 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:58.602371 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:58.602383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:58.604304 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
W20260806 16:12:58.604602 28141 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:58.598966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:48511 (local address 127.18.90.62:33657)
0806 16:12:58.599245 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:58.599249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:58.599265 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:12:58.599383 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:58.599385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:58.599426 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:58.599497 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:58.599909 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.600626 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.601401 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.601621 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.601696 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:58.604431 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.1:48511: BlockingRecv error: recv got EOF from 127.18.90.1:48511 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
Aug 06 16:12:58 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:12:58 dist-test-slave-0b58 krb5kdc[27071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.90.2@KRBTEST.COM for oryx/127.18.90.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:12:58.782013 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:58.776654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33657 (local address 127.18.90.2:37091)
0806 16:12:58.776955 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:58.776969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:58.777068 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:58.777293 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:58.777295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:58.777475 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:58.777580 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:58.777586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.778282 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.778284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:58.778825 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:58.778833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.779253 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.779256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:58.779326 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:58.779813 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:12:58.779825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:58.781851 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.90.62:33657: Server oryx/127.18.90.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
W20260806 16:12:58.782138 28141 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:58.776731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:37091 (local address 127.18.90.62:33657)
0806 16:12:58.776895 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:58.776901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:58.777039 (+ 138us) server_negotiation.cc:407] Connection header received
0806 16:12:58.777145 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:58.777147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:58.777195 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:58.777260 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:58.777675 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.778196 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.778919 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:58.779122 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:58.779193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:58.782003 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.90.2:37091: BlockingRecv error: recv got EOF from 127.18.90.2:37091 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:12:58.862361 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27661
I20260806 16:12:58.869094 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27816
I20260806 16:12:58.875358 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27949
I20260806 16:12:58.881383 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 27591
2026-08-06T16:12:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17986 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T16:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:58Z Disabled control of system clock
I20260806 16:12:58.913892 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:41501
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:41501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 16:12:59.026472 28146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:59.026741 28146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:59.026775 28146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:59.030937 28146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:59.031003 28146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:59.031019 28146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:59.031036 28146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:59.031052 28146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:59.035833 28146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:41501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:41501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:59.036922 28146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:59.038316 28146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:59.047397 28154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.047397 28151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.047537 28152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.047701 28146 server_base.cc:1034] running on GCE node
I20260806 16:12:59.048514 28146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:59.049609 28146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:59.050773 28146 hybrid_clock.cc:648] HybridClock initialized: now 1786032779050753 us; error 31 us; skew 500 ppm
I20260806 16:12:59.052799 28146 webserver.cc:460] Webserver started at http://127.18.90.62:46771/ using document root <none> and password file <none>
I20260806 16:12:59.053407 28146 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:59.053474 28146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:59.053742 28146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:59.055836 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "e34eb1416665483496308a980a67fe07"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "02db4b636b957ae8c542f17f6c817200"
server_key_iv: "96b46591a7345b0b068d24d259cd1e30"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.056473 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "e34eb1416665483496308a980a67fe07"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "02db4b636b957ae8c542f17f6c817200"
server_key_iv: "96b46591a7345b0b068d24d259cd1e30"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.060151 28146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:12:59.062638 28160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.063884 28146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:12:59.064024 28146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "e34eb1416665483496308a980a67fe07"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "02db4b636b957ae8c542f17f6c817200"
server_key_iv: "96b46591a7345b0b068d24d259cd1e30"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.064141 28146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:59.088907 28146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:59.089699 28146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:59.089929 28146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:59.097872 28146 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:41501
I20260806 16:12:59.097880 28212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:41501 every 8 connection(s)
I20260806 16:12:59.098877 28146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:59.100127 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28146
I20260806 16:12:59.100255 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:59.100579 18792 external_mini_cluster.cc:1428] Setting key 28f1614941bf50c2ef68db5546ab582a
I20260806 16:12:59.102671 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:59.109584 28213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07: Bootstrap starting.
I20260806 16:12:59.111265 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.101976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.18.90.62:41501)
0806 16:12:59.102666 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:59.102688 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:59.102744 (+ 56us) server_negotiation.cc:407] Connection header received
0806 16:12:59.103603 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:59.103626 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:59.104172 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:59.104546 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:59.104996 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.106496 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.107397 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.108102 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.108298 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.109195 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:59.109200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:59.109216 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:59.109720 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:59.109802 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:59.109828 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:59.109923 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:59.110063 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 16:12:59.110329 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
I20260806 16:12:59.112306 28213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:59.113175 28213 log.cc:824] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:59.115149 28213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07: No bootstrap required, opened a new log
I20260806 16:12:59.118758 28213 raft_consensus.cc:348] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34eb1416665483496308a980a67fe07" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41501 } }
I20260806 16:12:59.118985 28213 raft_consensus.cc:374] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:12:59.119026 28213 raft_consensus.cc:729] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34eb1416665483496308a980a67fe07, State: Initialized, Role: FOLLOWER
I20260806 16:12:59.119449 28213 consensus_queue.cc:260] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34eb1416665483496308a980a67fe07" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41501 } }
I20260806 16:12:59.119577 28213 raft_consensus.cc:388] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:12:59.119624 28213 raft_consensus.cc:482] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:12:59.119696 28213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:12:59.120666 28213 raft_consensus.cc:504] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34eb1416665483496308a980a67fe07" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41501 } }
I20260806 16:12:59.121033 28213 leader_election.cc:302] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e34eb1416665483496308a980a67fe07; no voters:
I20260806 16:12:59.121376 28213 leader_election.cc:288] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:12:59.121639 28218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:12:59.121857 28218 raft_consensus.cc:686] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [term 1 LEADER]: Becoming Leader. State: Replica: e34eb1416665483496308a980a67fe07, State: Running, Role: LEADER
I20260806 16:12:59.122166 28218 consensus_queue.cc:237] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34eb1416665483496308a980a67fe07" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41501 } }
I20260806 16:12:59.122684 28213 sys_catalog.cc:564] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:12:59.123945 28220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e34eb1416665483496308a980a67fe07. Latest consensus state: current_term: 1 leader_uuid: "e34eb1416665483496308a980a67fe07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34eb1416665483496308a980a67fe07" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41501 } } }
I20260806 16:12:59.123904 28219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e34eb1416665483496308a980a67fe07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34eb1416665483496308a980a67fe07" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 41501 } } }
I20260806 16:12:59.124061 28219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:59.124290 28220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07 [sys.catalog]: This master's current role is: LEADER
I20260806 16:12:59.124485 28227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:12:59.127758 28227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:12:59.133412 28227 catalog_manager.cc:1347] Generated new cluster ID: 3db9e663c98146d8bf82488fdea42b53
I20260806 16:12:59.133489 28227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:12:59.155548 28227 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:12:59.156648 28227 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:12:59.164949 28227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e34eb1416665483496308a980a67fe07: Generated new TSK 0
I20260806 16:12:59.165711 28227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:12:59.173887 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41501
--builtin_ntp_servers=127.18.90.20:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 16:12:59.289929 28237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:59.290270 28237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:59.290310 28237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:59.294327 28237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:59.294385 28237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:59.294437 28237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:12:59.299057 28237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:59.300159 28237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:59.301637 28237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:59.312314 28243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.312330 28245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.312870 28242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.312855 28237 server_base.cc:1034] running on GCE node
I20260806 16:12:59.313423 28237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:59.314410 28237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:59.315604 28237 hybrid_clock.cc:648] HybridClock initialized: now 1786032779315577 us; error 39 us; skew 500 ppm
I20260806 16:12:59.318137 28237 webserver.cc:460] Webserver started at http://127.18.90.1:37001/ using document root <none> and password file <none>
I20260806 16:12:59.319031 28237 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:59.319142 28237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:59.319412 28237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:59.322618 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "31098e5dba7c4bd0b9fbdcaca417b9a4"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "54d2c8685dc6b5b8ac536fad02d15ddd"
server_key_iv: "88efae40652083a5053089b4b1fa9fde"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.323480 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "31098e5dba7c4bd0b9fbdcaca417b9a4"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "54d2c8685dc6b5b8ac536fad02d15ddd"
server_key_iv: "88efae40652083a5053089b4b1fa9fde"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.329147 28237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 16:12:59.332837 28251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.334589 28237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:12:59.334761 28237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "31098e5dba7c4bd0b9fbdcaca417b9a4"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "54d2c8685dc6b5b8ac536fad02d15ddd"
server_key_iv: "88efae40652083a5053089b4b1fa9fde"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.334925 28237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:59.353777 28237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:59.354786 28237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:59.355028 28237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:59.355887 28237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:59.357081 28237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:59.357131 28237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.357170 28237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:59.357201 28237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.366266 28237 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:40699
I20260806 16:12:59.366291 28366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:40699 every 8 connection(s)
I20260806 16:12:59.367250 28237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:12:59.371470 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28237
I20260806 16:12:59.371565 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:12:59.371887 18792 external_mini_cluster.cc:1428] Setting key 7ef8e24277ec9f928679458728fb77f7
I20260806 16:12:59.373858 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41501
--builtin_ntp_servers=127.18.90.20:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 16:12:59.377362 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.369226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:44769 (local address 127.18.90.62:41501)
0806 16:12:59.369362 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:59.369366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:59.370511 (+ 1145us) server_negotiation.cc:407] Connection header received
0806 16:12:59.372087 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:59.372091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:59.372155 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:59.372285 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:59.373357 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.373901 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.375079 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.375351 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.375467 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.376745 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:59.376752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:59.376765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:59.376783 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:59.376809 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:59.376823 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:59.376904 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:59.377050 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 16:12:59.377130 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:12:59.378082 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.369696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:41501 (local address 127.18.90.1:44769)
0806 16:12:59.370367 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:59.370408 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:59.371816 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:59.372472 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:59.372478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:59.372562 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:59.373211 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:59.373226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.374034 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.374040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:59.374941 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:59.374950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.375543 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.375550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:59.375770 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.376477 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:59.376478 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:59.376519 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:59.376582 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:59.376910 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:59.376917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:59.376921 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:12:59.377014 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 16:12:59.377243 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260806 16:12:59.379383 28367 heartbeater.cc:347] Connected to a master server at 127.18.90.62:41501
I20260806 16:12:59.379655 28367 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:59.380373 28367 heartbeater.cc:511] Master 127.18.90.62:41501 requested a full tablet report, sending...
I20260806 16:12:59.382073 28177 ts_manager.cc:194] Registered new tserver with Master: 31098e5dba7c4bd0b9fbdcaca417b9a4 (127.18.90.1:40699)
I20260806 16:12:59.383246 28177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.1:44769
W20260806 16:12:59.488783 28370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:59.489035 28370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:59.489073 28370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:59.493085 28370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:59.493139 28370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:59.493229 28370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:12:59.497938 28370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:59.499032 28370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:59.500458 28370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:59.510311 28375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.510401 28376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.510422 28370 server_base.cc:1034] running on GCE node
W20260806 16:12:59.510490 28378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.511020 28370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:59.511710 28370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:59.512900 28370 hybrid_clock.cc:648] HybridClock initialized: now 1786032779512891 us; error 31 us; skew 500 ppm
I20260806 16:12:59.514701 28370 webserver.cc:460] Webserver started at http://127.18.90.2:43959/ using document root <none> and password file <none>
I20260806 16:12:59.515298 28370 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:59.515393 28370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:59.515618 28370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:59.517740 28370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "f101ed4fa1cc4a78a082553c1a59f72c"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "d742b5f6634951774617b2395ec94097"
server_key_iv: "72e1e90e2f98ff8a56a1db455d5573d9"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.518327 28370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "f101ed4fa1cc4a78a082553c1a59f72c"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "d742b5f6634951774617b2395ec94097"
server_key_iv: "72e1e90e2f98ff8a56a1db455d5573d9"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.522122 28370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:12:59.524945 28384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.526305 28370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:59.526432 28370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "f101ed4fa1cc4a78a082553c1a59f72c"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "d742b5f6634951774617b2395ec94097"
server_key_iv: "72e1e90e2f98ff8a56a1db455d5573d9"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.526556 28370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:59.542160 28370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:59.542863 28370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:59.543064 28370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:59.543677 28370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:59.544734 28370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:59.544782 28370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.544852 28370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:59.544888 28370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.553998 28370 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:35001
I20260806 16:12:59.554021 28498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:35001 every 8 connection(s)
I20260806 16:12:59.554965 28370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:12:59.561039 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28370
I20260806 16:12:59.561164 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:12:59.561468 18792 external_mini_cluster.cc:1428] Setting key fd689fdc49637b5d6c3d981374e36abd
I20260806 16:12:59.566020 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41501
--builtin_ntp_servers=127.18.90.20:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 16:12:59.571288 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.558937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:59545 (local address 127.18.90.62:41501)
0806 16:12:59.560454 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:59.560458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:59.562300 (+ 1842us) server_negotiation.cc:407] Connection header received
0806 16:12:59.564097 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:59.564101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:59.564156 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:59.564205 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:59.565903 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.566602 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.568061 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.568512 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.568624 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.570231 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:59.570236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:59.570240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:59.570258 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:59.570286 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:59.570297 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:59.570376 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:59.570982 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 16:12:59.571074 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404}
I20260806 16:12:59.573112 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.557653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:41501 (local address 127.18.90.2:59545)
0806 16:12:59.562125 (+ 4472us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:59.562183 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:59.563853 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:59.564640 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:59.564649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:59.564758 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:59.565411 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:59.565429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.566736 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.566740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:59.567933 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:59.567948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.568687 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.568694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:59.568943 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.569893 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:59.569898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:59.569957 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:59.570049 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:59.570765 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:59.570779 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:59.570786 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:12:59.571495 (+ 709us) client_negotiation.cc:240] Negotiation successful
0806 16:12:59.571872 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2851,"thread_start_us":152,"threads_started":1}
I20260806 16:12:59.574451 28499 heartbeater.cc:347] Connected to a master server at 127.18.90.62:41501
I20260806 16:12:59.574734 28499 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:59.575287 28499 heartbeater.cc:511] Master 127.18.90.62:41501 requested a full tablet report, sending...
I20260806 16:12:59.576506 28177 ts_manager.cc:194] Registered new tserver with Master: f101ed4fa1cc4a78a082553c1a59f72c (127.18.90.2:35001)
I20260806 16:12:59.577207 28177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.2:59545
W20260806 16:12:59.681968 28502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:59.682238 28502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:59.682274 28502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:59.686367 28502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:59.686426 28502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:59.686479 28502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:12:59.691172 28502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:41501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:59.692241 28502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:59.693670 28502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:59.703060 28507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.703053 28508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.703222 28502 server_base.cc:1034] running on GCE node
W20260806 16:12:59.703424 28510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.703902 28502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:59.704583 28502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:59.705742 28502 hybrid_clock.cc:648] HybridClock initialized: now 1786032779705707 us; error 46 us; skew 500 ppm
I20260806 16:12:59.707837 28502 webserver.cc:460] Webserver started at http://127.18.90.3:32993/ using document root <none> and password file <none>
I20260806 16:12:59.708597 28502 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:59.708665 28502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:59.708889 28502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:59.710963 28502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "14080693cc17442599373f9b648742b3"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "d20451056d94ce506702769a6100ce20"
server_key_iv: "f5e29e70ab354ceae806d27e15f48ae0"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.711530 28502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "14080693cc17442599373f9b648742b3"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "d20451056d94ce506702769a6100ce20"
server_key_iv: "f5e29e70ab354ceae806d27e15f48ae0"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.715286 28502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:12:59.717937 28516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.719234 28502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:12:59.719380 28502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "14080693cc17442599373f9b648742b3"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "d20451056d94ce506702769a6100ce20"
server_key_iv: "f5e29e70ab354ceae806d27e15f48ae0"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.719507 28502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:59.732955 28502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:59.733675 28502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:59.733888 28502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:59.734494 28502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:12:59.735455 28502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:12:59.735524 28502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.735599 28502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:12:59.735643 28502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.744783 28502 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:38083
I20260806 16:12:59.744863 28630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:38083 every 8 connection(s)
I20260806 16:12:59.745731 28502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:12:59.754029 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28502
I20260806 16:12:59.754154 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:12:59.754488 18792 external_mini_cluster.cc:1428] Setting key f82e7b2f47bee47a4d285cb04b2ae40a
I20260806 16:12:59.754846 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.747613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:43191 (local address 127.18.90.62:41501)
0806 16:12:59.747758 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:59.747761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:59.748795 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 16:12:59.750174 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:59.750177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:59.750227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:59.750306 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:59.751223 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.751760 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.752899 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.753117 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.753202 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.754315 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:59.754318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:59.754320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:59.754334 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:59.754354 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:59.754361 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:59.754422 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:59.754620 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 16:12:59.754693 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:12:59.755837 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.747959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:41501 (local address 127.18.90.3:43191)
0806 16:12:59.748658 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0806 16:12:59.748700 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:12:59.749970 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:12:59.750470 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:12:59.750477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:12:59.750559 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:12:59.751058 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:59.751071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.751875 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.751877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:59.752778 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:12:59.752795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:12:59.753255 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:12:59.753258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:12:59.753403 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:12:59.753983 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:12:59.753985 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:12:59.754044 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:12:59.754134 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:12:59.754450 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:12:59.754460 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:12:59.754466 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:12:59.754602 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 16:12:59.754920 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260806 16:12:59.757041 28631 heartbeater.cc:347] Connected to a master server at 127.18.90.62:41501
I20260806 16:12:59.757263 28631 heartbeater.cc:464] Registering TS with master...
I20260806 16:12:59.757781 28631 heartbeater.cc:511] Master 127.18.90.62:41501 requested a full tablet report, sending...
I20260806 16:12:59.758776 28176 ts_manager.cc:194] Registered new tserver with Master: 14080693cc17442599373f9b648742b3 (127.18.90.3:38083)
I20260806 16:12:59.759374 28176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.3:43191
I20260806 16:12:59.768400 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 16:12:59.771088 28216 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:12:59.770290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.18.90.62:41501)
0806 16:12:59.770393 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:59.770397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:59.770616 (+ 219us) server_negotiation.cc:407] Connection header received
0806 16:12:59.770708 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:59.770711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:59.770753 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:59.770822 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:59.770990 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44024: BlockingRecv error: recv got EOF from 127.0.0.1:44024 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 16:12:59.771265 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28237
I20260806 16:12:59.778625 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28370
I20260806 16:12:59.784982 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28502
I20260806 16:12:59.791034 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28146
2026-08-06T16:12:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (908 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T16:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:12:59Z Disabled control of system clock
I20260806 16:12:59.821816 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:37739
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 16:12:59.933311 28642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:12:59.933598 28642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:12:59.933636 28642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:12:59.937590 28642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:12:59.937649 28642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:12:59.937665 28642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:12:59.937681 28642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:12:59.937696 28642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:12:59.942369 28642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:37739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:12:59.943437 28642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:12:59.944813 28642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:12:59.953104 28648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.953142 28647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:12:59.953104 28650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:12:59.953366 28642 server_base.cc:1034] running on GCE node
I20260806 16:12:59.954053 28642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:12:59.955226 28642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:12:59.956413 28642 hybrid_clock.cc:648] HybridClock initialized: now 1786032779956391 us; error 44 us; skew 500 ppm
I20260806 16:12:59.958380 28642 webserver.cc:460] Webserver started at http://127.18.90.62:36839/ using document root <none> and password file <none>
I20260806 16:12:59.958977 28642 fs_manager.cc:359] Metadata directory not provided
I20260806 16:12:59.959074 28642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:12:59.959331 28642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:12:59.961483 28642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "174ff6d3b2304cea9db45892de26edbe"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "c062c818e9957217bc62747841df064b"
server_key_iv: "cf09e75f2ac2e007247b849364b95d5b"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.962059 28642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "174ff6d3b2304cea9db45892de26edbe"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "c062c818e9957217bc62747841df064b"
server_key_iv: "cf09e75f2ac2e007247b849364b95d5b"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.965732 28642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:12:59.968113 28656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:12:59.969359 28642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:12:59.969503 28642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "174ff6d3b2304cea9db45892de26edbe"
format_stamp: "Formatted at 2026-08-06 16:12:59 on dist-test-slave-0b58"
server_key: "c062c818e9957217bc62747841df064b"
server_key_iv: "cf09e75f2ac2e007247b849364b95d5b"
server_key_version: "encryptionkey@0"
I20260806 16:12:59.969621 28642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:12:59.974193 28642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:12:59.974843 28642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:12:59.975028 28642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:12:59.981766 28642 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:37739
I20260806 16:12:59.981777 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:37739 every 8 connection(s)
I20260806 16:12:59.982697 28642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:12:59.985971 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:12:59.987739 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28642
I20260806 16:12:59.987867 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:12:59.988148 18792 external_mini_cluster.cc:1428] Setting key ea48e232c3bf583d96485e526bf52c61
I20260806 16:12:59.992512 28709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe: Bootstrap starting.
I20260806 16:12:59.994504 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:12:59.989653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.18.90.62:37739)
0806 16:12:59.990352 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0806 16:12:59.990376 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 16:12:59.990421 (+ 45us) server_negotiation.cc:407] Connection header received
0806 16:12:59.991382 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:12:59.991407 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:12:59.991854 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:12:59.992212 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:12:59.993037 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:12:59.993043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:12:59.993058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:12:59.993069 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:12:59.993138 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:12:59.993161 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:12:59.993304 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 16:12:59.993369 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 16:12:59.993603 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260806 16:12:59.995210 28709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe: Neither blocks nor log segments found. Creating new log.
I20260806 16:12:59.995996 28709 log.cc:824] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe: Log is configured to *not* fsync() on all Append() calls
I20260806 16:12:59.997928 28709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe: No bootstrap required, opened a new log
I20260806 16:13:00.001513 28709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174ff6d3b2304cea9db45892de26edbe" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37739 } }
I20260806 16:13:00.001801 28709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:00.001885 28709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174ff6d3b2304cea9db45892de26edbe, State: Initialized, Role: FOLLOWER
I20260806 16:13:00.002296 28709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174ff6d3b2304cea9db45892de26edbe" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37739 } }
I20260806 16:13:00.002435 28709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:13:00.002501 28709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:13:00.002604 28709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:00.003517 28709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174ff6d3b2304cea9db45892de26edbe" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37739 } }
I20260806 16:13:00.003865 28709 leader_election.cc:302] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 174ff6d3b2304cea9db45892de26edbe; no voters:
I20260806 16:13:00.004179 28709 leader_election.cc:288] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:13:00.004329 28714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:13:00.004554 28714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [term 1 LEADER]: Becoming Leader. State: Replica: 174ff6d3b2304cea9db45892de26edbe, State: Running, Role: LEADER
I20260806 16:13:00.004940 28714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174ff6d3b2304cea9db45892de26edbe" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37739 } }
I20260806 16:13:00.005481 28709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:13:00.006603 28716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 174ff6d3b2304cea9db45892de26edbe. Latest consensus state: current_term: 1 leader_uuid: "174ff6d3b2304cea9db45892de26edbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174ff6d3b2304cea9db45892de26edbe" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37739 } } }
I20260806 16:13:00.006739 28716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [sys.catalog]: This master's current role is: LEADER
I20260806 16:13:00.007138 28723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:13:00.007114 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "174ff6d3b2304cea9db45892de26edbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174ff6d3b2304cea9db45892de26edbe" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37739 } } }
I20260806 16:13:00.007268 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe [sys.catalog]: This master's current role is: LEADER
I20260806 16:13:00.009982 28723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:13:00.015187 28723 catalog_manager.cc:1347] Generated new cluster ID: 4c6f9e60c60645c8b8d6229ab6a7317d
I20260806 16:13:00.015255 28723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:13:00.026907 28723 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:13:00.027235 28723 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:13:00.031520 28723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 174ff6d3b2304cea9db45892de26edbe: Generated new TSK 0
I20260806 16:13:00.032333 28723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:13:00.039143 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37739
--builtin_ntp_servers=127.18.90.20:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 16:13:00.153244 28734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:00.153497 28734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:00.153537 28734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:00.157575 28734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:00.157634 28734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:00.157744 28734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:13:00.162663 28734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:00.163815 28734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:00.165242 28734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:00.175416 28742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:00.175439 28739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:00.175603 28734 server_base.cc:1034] running on GCE node
W20260806 16:13:00.175421 28740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:00.176142 28734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:00.176831 28734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:00.178025 28734 hybrid_clock.cc:648] HybridClock initialized: now 1786032780178010 us; error 46 us; skew 500 ppm
I20260806 16:13:00.179946 28734 webserver.cc:460] Webserver started at http://127.18.90.1:43991/ using document root <none> and password file <none>
I20260806 16:13:00.180624 28734 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:00.180727 28734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:00.180971 28734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:00.183079 28734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "b0a37fcda3254238bc0a026ae86e48b2"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "3bec833a8403e878f87da3cc7b9519ff"
server_key_iv: "b4fd0e7f4398a165e3e5e35f200f2b91"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.183681 28734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "b0a37fcda3254238bc0a026ae86e48b2"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "3bec833a8403e878f87da3cc7b9519ff"
server_key_iv: "b4fd0e7f4398a165e3e5e35f200f2b91"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.187521 28734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:13:00.190110 28748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.191308 28734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:13:00.191453 28734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "b0a37fcda3254238bc0a026ae86e48b2"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "3bec833a8403e878f87da3cc7b9519ff"
server_key_iv: "b4fd0e7f4398a165e3e5e35f200f2b91"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.191576 28734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:00.196199 28734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:00.196851 28734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:00.197036 28734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:00.197582 28734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:00.198490 28734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:00.198560 28734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.198621 28734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:00.198673 28734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.207305 28734 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:40391
I20260806 16:13:00.207341 28861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:40391 every 8 connection(s)
I20260806 16:13:00.208225 28734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:13:00.214272 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:00.210066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:59655 (local address 127.18.90.62:37739)
0806 16:13:00.210215 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:00.210219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:00.211280 (+ 1061us) server_negotiation.cc:407] Connection header received
0806 16:13:00.212629 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:00.212632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:00.212671 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:00.212746 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:00.213873 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:00.213876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:00.213878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:13:00.213889 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:00.213917 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:13:00.213924 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:00.214001 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:00.214127 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 16:13:00.214164 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:13:00.215150 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:00.210485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:37739 (local address 127.18.90.1:59655)
0806 16:13:00.211136 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:00.211176 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:00.212439 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:00.212916 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:00.212923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:00.213010 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:00.213552 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:13:00.213554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:00.213611 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:13:00.213714 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:00.214002 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:00.214010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:00.214014 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:13:00.214149 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 16:13:00.214365 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260806 16:13:00.216207 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28734
I20260806 16:13:00.216356 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:13:00.216526 28862 heartbeater.cc:347] Connected to a master server at 127.18.90.62:37739
I20260806 16:13:00.216759 18792 external_mini_cluster.cc:1428] Setting key 11c6a910ae29c252d25789e651bf33d5
I20260806 16:13:00.216776 28862 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:00.217242 28862 heartbeater.cc:511] Master 127.18.90.62:37739 requested a full tablet report, sending...
I20260806 16:13:00.218647 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37739
--builtin_ntp_servers=127.18.90.20:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 16:13:00.218775 28673 ts_manager.cc:194] Registered new tserver with Master: b0a37fcda3254238bc0a026ae86e48b2 (127.18.90.1:40391)
W20260806 16:13:00.334789 28865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:00.335067 28865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:00.335109 28865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:00.339226 28865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:00.339293 28865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:00.339388 28865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:13:00.344174 28865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:00.345396 28865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:00.346827 28865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:00.356444 28873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:00.356433 28870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:00.356492 28865 server_base.cc:1034] running on GCE node
W20260806 16:13:00.356438 28871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:00.357084 28865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:00.357728 28865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:00.358917 28865 hybrid_clock.cc:648] HybridClock initialized: now 1786032780358887 us; error 43 us; skew 500 ppm
I20260806 16:13:00.360767 28865 webserver.cc:460] Webserver started at http://127.18.90.2:45557/ using document root <none> and password file <none>
I20260806 16:13:00.361347 28865 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:00.361444 28865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:00.361677 28865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:00.363747 28865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "dc90896489f94f139505ce3c618c5634"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "7672bb50e7b7657e935a21fcc7a5486e"
server_key_iv: "d468607dd158468f2d3e6159750885f9"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.364403 28865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "dc90896489f94f139505ce3c618c5634"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "7672bb50e7b7657e935a21fcc7a5486e"
server_key_iv: "d468607dd158468f2d3e6159750885f9"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.368108 28865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:13:00.370611 28879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.371855 28865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:13:00.371987 28865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "dc90896489f94f139505ce3c618c5634"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "7672bb50e7b7657e935a21fcc7a5486e"
server_key_iv: "d468607dd158468f2d3e6159750885f9"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.372105 28865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:00.376864 28865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:00.377550 28865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:00.377737 28865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:00.378276 28865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:00.379176 28865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:00.379251 28865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.379317 28865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:00.379369 28865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.388058 28865 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:34023
I20260806 16:13:00.388078 28994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:34023 every 8 connection(s)
I20260806 16:13:00.389077 28865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:13:00.395272 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:00.390864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:46639 (local address 127.18.90.62:37739)
0806 16:13:00.391006 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:00.391011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:00.392001 (+ 990us) server_negotiation.cc:407] Connection header received
0806 16:13:00.393429 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:00.393432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:00.393481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:00.393589 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:00.394766 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:00.394769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:00.394772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:13:00.394785 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:00.394810 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:13:00.394824 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:00.394926 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:00.395084 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 16:13:00.395132 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:13:00.396174 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:00.391207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:37739 (local address 127.18.90.2:46639)
0806 16:13:00.391855 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:00.391895 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:00.393194 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:00.393761 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:00.393767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:00.393857 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:00.394489 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:13:00.394491 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:00.394549 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:13:00.394619 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:00.394919 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:00.394928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:00.394934 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:13:00.395116 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 16:13:00.395368 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260806 16:13:00.396888 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28865
I20260806 16:13:00.397018 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:13:00.397313 18792 external_mini_cluster.cc:1428] Setting key 5c58917acd9d4f54b9700bd6ed8f6244
I20260806 16:13:00.397508 28995 heartbeater.cc:347] Connected to a master server at 127.18.90.62:37739
I20260806 16:13:00.397745 28995 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:00.398231 28995 heartbeater.cc:511] Master 127.18.90.62:37739 requested a full tablet report, sending...
I20260806 16:13:00.399339 28673 ts_manager.cc:194] Registered new tserver with Master: dc90896489f94f139505ce3c618c5634 (127.18.90.2:34023)
I20260806 16:13:00.399374 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37739
--builtin_ntp_servers=127.18.90.20:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 16:13:00.512723 28998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:00.513023 28998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:00.513105 28998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:00.517151 28998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:00.517235 28998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:00.517323 28998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:13:00.522095 28998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:00.523239 28998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:00.524717 28998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:00.534541 29003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:00.534562 29006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:00.534561 29004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:00.534772 28998 server_base.cc:1034] running on GCE node
I20260806 16:13:00.535281 28998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:00.535951 28998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:00.537139 28998 hybrid_clock.cc:648] HybridClock initialized: now 1786032780537103 us; error 59 us; skew 500 ppm
I20260806 16:13:00.539041 28998 webserver.cc:460] Webserver started at http://127.18.90.3:41661/ using document root <none> and password file <none>
I20260806 16:13:00.539647 28998 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:00.539742 28998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:00.539983 28998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:00.542586 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "66519cf8bd0641f8ad1e9227f2f4acd8"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "69d41d44e1e3470cf269d39570ba6786"
server_key_iv: "055651e72e15397682cc55c315f31a04"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.543179 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "66519cf8bd0641f8ad1e9227f2f4acd8"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "69d41d44e1e3470cf269d39570ba6786"
server_key_iv: "055651e72e15397682cc55c315f31a04"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.546953 28998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:13:00.549540 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.550822 28998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:13:00.550967 28998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "66519cf8bd0641f8ad1e9227f2f4acd8"
format_stamp: "Formatted at 2026-08-06 16:13:00 on dist-test-slave-0b58"
server_key: "69d41d44e1e3470cf269d39570ba6786"
server_key_iv: "055651e72e15397682cc55c315f31a04"
server_key_version: "encryptionkey@0"
I20260806 16:13:00.551088 28998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:00.555612 28998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:00.556206 28998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:00.556427 28998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:00.556972 28998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:00.557886 28998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:00.557956 28998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.558027 28998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:00.558079 28998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:00.566846 28998 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:33615
I20260806 16:13:00.566866 29126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:33615 every 8 connection(s)
I20260806 16:13:00.567790 28998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:13:00.573874 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:00.569693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:50673 (local address 127.18.90.62:37739)
0806 16:13:00.569852 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:00.569857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:00.570968 (+ 1111us) server_negotiation.cc:407] Connection header received
0806 16:13:00.572234 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:00.572237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:00.572319 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:00.572399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:00.573489 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:00.573492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:00.573494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:13:00.573505 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:00.573525 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:13:00.573533 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:00.573609 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:00.573728 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 16:13:00.573764 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:13:00.574774 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:00.570163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:37739 (local address 127.18.90.3:50673)
0806 16:13:00.570820 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:00.570861 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:00.572036 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:00.572561 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:00.572567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:00.572651 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:00.573251 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:13:00.573252 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:00.573293 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:13:00.573358 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:00.573607 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:00.573613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:00.573617 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:13:00.573751 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 16:13:00.573966 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260806 16:13:00.575896 29127 heartbeater.cc:347] Connected to a master server at 127.18.90.62:37739
I20260806 16:13:00.576301 29127 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:00.576566 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 28998
I20260806 16:13:00.576696 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:13:00.576800 29127 heartbeater.cc:511] Master 127.18.90.62:37739 requested a full tablet report, sending...
I20260806 16:13:00.577037 18792 external_mini_cluster.cc:1428] Setting key 43fe376ecbc96d26d843f9bf5a904dac
I20260806 16:13:00.578012 28672 ts_manager.cc:194] Registered new tserver with Master: 66519cf8bd0641f8ad1e9227f2f4acd8 (127.18.90.3:33615)
I20260806 16:13:00.579185 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 16:13:00.582111 28712 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:13:00.581146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.18.90.62:37739)
0806 16:13:00.581243 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:00.581247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:00.581499 (+ 252us) server_negotiation.cc:407] Connection header received
0806 16:13:00.581635 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:00.581638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:00.581680 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:00.581762 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:00.581989 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45426: BlockingRecv error: recv got EOF from 127.0.0.1:45426 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 16:13:00.582441 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28734
I20260806 16:13:00.589831 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28865
I20260806 16:13:00.595688 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28998
I20260806 16:13:00.601552 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 28642
2026-08-06T16:13:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (810 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:13:00 dist-test-slave-0b58 krb5kdc[29137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:13:00 dist-test-slave-0b58 krb5kdc[29137](info): set up 2 sockets
Aug 06 16:13:00 dist-test-slave-0b58 krb5kdc[29137](info): commencing operation
krb5kdc: starting...
W20260806 16:13:01.641526 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:13:01 dist-test-slave-0b58 krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:13:01.767954 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36667
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:01.880697 29150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:01.881049 29150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:01.881153 29150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:01.885239 29150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:13:01.885321 29150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:01.885361 29150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:01.885419 29150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:13:01.885460 29150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:13:01.890730 29150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:01.891857 29150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:01.892998 29150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:01.901746 29155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:01.901741 29158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:01.901748 29156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:01.902153 29150 server_base.cc:1034] running on GCE node
I20260806 16:13:01.902894 29150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:01.904076 29150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:01.905280 29150 hybrid_clock.cc:648] HybridClock initialized: now 1786032781905263 us; error 40 us; skew 500 ppm
Aug 06 16:13:01 dist-test-slave-0b58 krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032781, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:01.911585 29150 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:13:01.912626 29150 webserver.cc:460] Webserver started at http://127.18.90.62:46513/ using document root <none> and password file <none>
I20260806 16:13:01.913213 29150 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:01.913309 29150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:01.913546 29150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:01.915624 29150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "fef8c3d4f8ba413a90d4b399dc013719"
format_stamp: "Formatted at 2026-08-06 16:13:01 on dist-test-slave-0b58"
server_key: "f7e9c3c2875a38d5829d79e41437eb09"
server_key_iv: "cdd9e952654ead766c4d801018f7e69a"
server_key_version: "encryptionkey@0"
I20260806 16:13:01.916199 29150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "fef8c3d4f8ba413a90d4b399dc013719"
format_stamp: "Formatted at 2026-08-06 16:13:01 on dist-test-slave-0b58"
server_key: "f7e9c3c2875a38d5829d79e41437eb09"
server_key_iv: "cdd9e952654ead766c4d801018f7e69a"
server_key_version: "encryptionkey@0"
I20260806 16:13:01.919899 29150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:13:01.922465 29165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:01.923842 29150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:13:01.923993 29150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "fef8c3d4f8ba413a90d4b399dc013719"
format_stamp: "Formatted at 2026-08-06 16:13:01 on dist-test-slave-0b58"
server_key: "f7e9c3c2875a38d5829d79e41437eb09"
server_key_iv: "cdd9e952654ead766c4d801018f7e69a"
server_key_version: "encryptionkey@0"
I20260806 16:13:01.924101 29150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:01.957245 29150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:01.960835 29150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:01.961053 29150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:01.968770 29150 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:36667
I20260806 16:13:01.969051 29217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:36667 every 8 connection(s)
I20260806 16:13:01.969828 29150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:13:01.972743 29218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:01.974849 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 29150
I20260806 16:13:01.974982 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:13:01.975302 18792 external_mini_cluster.cc:1428] Setting key ddc3e9e8ad7012ffa8b753ce3e1dc123
I20260806 16:13:01.979753 29218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719: Bootstrap starting.
I20260806 16:13:01.982432 29218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:01.983357 29218 log.cc:824] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:13:01 dist-test-slave-0b58 krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:13:01.985687 29218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719: No bootstrap required, opened a new log
I20260806 16:13:01.990494 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:01.976780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.18.90.62:36667)
0806 16:13:01.977202 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:01.977219 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:01.977257 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:13:01.977911 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:01.977927 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:01.978254 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:01.978527 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:01.979453 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:01.980539 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:01.981737 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:01.982210 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:01.982335 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:01.985171 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:01.985178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:01.985194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:01.985225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:01.987236 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:01.987668 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:01.987674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:01.987680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:01.987834 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:01.988104 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:01.988108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:01.988110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:01.988742 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:01.988941 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:01.989154 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 16:13:01.989467 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":148,"threads_started":1}
I20260806 16:13:01.990618 29218 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8c3d4f8ba413a90d4b399dc013719" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36667 } }
I20260806 16:13:01.990901 29218 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:01.990975 29218 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef8c3d4f8ba413a90d4b399dc013719, State: Initialized, Role: FOLLOWER
I20260806 16:13:01.991463 29218 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8c3d4f8ba413a90d4b399dc013719" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36667 } }
I20260806 16:13:01.991636 29218 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:13:01.991706 29218 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:13:01.991801 29218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:01.992972 29218 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8c3d4f8ba413a90d4b399dc013719" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36667 } }
I20260806 16:13:01.993325 29218 leader_election.cc:302] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef8c3d4f8ba413a90d4b399dc013719; no voters:
I20260806 16:13:01.993655 29218 leader_election.cc:288] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:13:01.993795 29223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:13:01.994029 29223 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [term 1 LEADER]: Becoming Leader. State: Replica: fef8c3d4f8ba413a90d4b399dc013719, State: Running, Role: LEADER
I20260806 16:13:01.994668 29223 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8c3d4f8ba413a90d4b399dc013719" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36667 } }
I20260806 16:13:01.994966 29218 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:13:01.996421 29225 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef8c3d4f8ba413a90d4b399dc013719. Latest consensus state: current_term: 1 leader_uuid: "fef8c3d4f8ba413a90d4b399dc013719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8c3d4f8ba413a90d4b399dc013719" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36667 } } }
I20260806 16:13:01.996469 29224 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef8c3d4f8ba413a90d4b399dc013719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8c3d4f8ba413a90d4b399dc013719" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36667 } } }
I20260806 16:13:01.996558 29225 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [sys.catalog]: This master's current role is: LEADER
I20260806 16:13:01.996585 29224 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719 [sys.catalog]: This master's current role is: LEADER
I20260806 16:13:01.997006 29234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:13:01.999727 29234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:13:02.005162 29234 catalog_manager.cc:1347] Generated new cluster ID: 3a748c828a114309a645443432cd1436
I20260806 16:13:02.005242 29234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:13:02.017026 29234 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:13:02.018290 29234 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:13:02.041908 29234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719: Generated new TSK 0
I20260806 16:13:02.042757 29234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:13:02.083844 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36667
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:02.199316 29244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:02.199584 29244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:02.199628 29244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:02.203559 29244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:02.203616 29244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:02.203673 29244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:13:02.208825 29244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:02.209796 29244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:02.210834 29244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:02.220023 29250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:02.220052 29249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:02.220348 29252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:02.220506 29244 server_base.cc:1034] running on GCE node
I20260806 16:13:02.220917 29244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:02.221518 29244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:02.222675 29244 hybrid_clock.cc:648] HybridClock initialized: now 1786032782222643 us; error 43 us; skew 500 ppm
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032782, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:02.226951 29244 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:13:02.228128 29244 webserver.cc:460] Webserver started at http://127.18.90.1:34419/ using document root <none> and password file <none>
I20260806 16:13:02.228912 29244 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:02.229010 29244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:02.229274 29244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:02.232005 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "9c91d4adc347473485c643d6f4cdb2cb"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "45729e9ac83c18dc7a401a8d6f1d0801"
server_key_iv: "122a55450086675e2411d1c486561d25"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.232614 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "9c91d4adc347473485c643d6f4cdb2cb"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "45729e9ac83c18dc7a401a8d6f1d0801"
server_key_iv: "122a55450086675e2411d1c486561d25"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.236328 29244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:13:02.238832 29259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.240031 29244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:13:02.240167 29244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "9c91d4adc347473485c643d6f4cdb2cb"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "45729e9ac83c18dc7a401a8d6f1d0801"
server_key_iv: "122a55450086675e2411d1c486561d25"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.240305 29244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:02.253608 29244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:02.256703 29244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:02.256867 29244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:02.257398 29244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:02.258241 29244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:02.258292 29244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.258359 29244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:02.258391 29244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.267403 29244 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:37145
I20260806 16:13:02.267426 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:37145 every 8 connection(s)
I20260806 16:13:02.268484 29244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:13:02.271183 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 29244
I20260806 16:13:02.271337 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:13:02.271687 18792 external_mini_cluster.cc:1428] Setting key 6f58b4b0e21632f6506a30a74537222b
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032782, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:13:02.284106 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.270361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:51107 (local address 127.18.90.62:36667)
0806 16:13:02.270500 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.270504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.271461 (+ 957us) server_negotiation.cc:407] Connection header received
0806 16:13:02.273152 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.273156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.273209 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.273303 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:02.274936 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.275656 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.276902 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.277313 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.277440 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.279887 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:02.279893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:02.279898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:02.279930 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:02.281715 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.282240 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.282243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.282245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.282335 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.282739 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.282744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.282746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.282995 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:02.283079 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.283841 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.283971 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:13:02.285143 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.270675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36667 (local address 127.18.90.1:51107)
0806 16:13:02.271229 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.271299 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.272896 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.273502 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.273512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.274055 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:02.274782 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.274805 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.275797 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.275800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.276776 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.276786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.277465 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.277468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.277611 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.278240 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:13:02.278257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:02.279700 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:02.281850 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:02.281855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:02.281867 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:02.282126 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:02.282430 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:02.282432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:02.282433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:02.282566 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:02.283091 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:02.283097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:02.283544 (+ 447us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.283794 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.284128 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:13:02.286589 29374 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36667
I20260806 16:13:02.286870 29374 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:02.287400 29374 heartbeater.cc:511] Master 127.18.90.62:36667 requested a full tablet report, sending...
I20260806 16:13:02.288970 29181 ts_manager.cc:194] Registered new tserver with Master: 9c91d4adc347473485c643d6f4cdb2cb (127.18.90.1:37145)
I20260806 16:13:02.290657 29181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:51107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:13:02.302819 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36667
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:02.414832 29379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:02.415096 29379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:02.415139 29379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:02.419190 29379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:02.419246 29379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:02.419339 29379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:13:02.424638 29379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:02.425695 29379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:02.426731 29379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:02.436390 29385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:02.436473 29379 server_base.cc:1034] running on GCE node
W20260806 16:13:02.436398 29384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:02.436398 29387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:02.437052 29379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:02.437678 29379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:02.438885 29379 hybrid_clock.cc:648] HybridClock initialized: now 1786032782438871 us; error 31 us; skew 500 ppm
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032782, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:02.441648 29379 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:13:02.442472 29379 webserver.cc:460] Webserver started at http://127.18.90.2:37433/ using document root <none> and password file <none>
I20260806 16:13:02.443037 29379 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:02.443122 29379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:02.443297 29379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:02.445359 29379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "b6c24d6c8a33c70b25d79d1a5fb14659"
server_key_iv: "4e76c0f4ee540a4b9bd1e0c6fde692d6"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.445880 29379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "b6c24d6c8a33c70b25d79d1a5fb14659"
server_key_iv: "4e76c0f4ee540a4b9bd1e0c6fde692d6"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.449540 29379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:13:02.451902 29394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.453182 29379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:13:02.453289 29379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "b6c24d6c8a33c70b25d79d1a5fb14659"
server_key_iv: "4e76c0f4ee540a4b9bd1e0c6fde692d6"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.453374 29379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:02.464532 29379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:02.467769 29379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:02.467921 29379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:02.468521 29379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:02.469295 29379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:02.469342 29379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.469383 29379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:02.469403 29379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.477941 29379 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:33853
I20260806 16:13:02.477964 29508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:33853 every 8 connection(s)
I20260806 16:13:02.478878 29379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:13:02.479830 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 29379
I20260806 16:13:02.479965 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:13:02.480320 18792 external_mini_cluster.cc:1428] Setting key 9ce86746a019ed210ffdb730759b6c73
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032782, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:13:02.494987 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.480803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:35877 (local address 127.18.90.62:36667)
0806 16:13:02.480947 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.480950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.481983 (+ 1033us) server_negotiation.cc:407] Connection header received
0806 16:13:02.484017 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.484020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.484066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.484193 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:02.485419 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.485986 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.487865 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.488082 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.488156 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.491054 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:02.491058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:02.491060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:02.491086 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:02.492659 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.493171 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.493174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.493176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.493263 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.493606 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.493612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.493616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.493878 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:02.493991 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.494652 (+ 661us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.494781 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:13:02.495944 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.481254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36667 (local address 127.18.90.2:35877)
0806 16:13:02.481806 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.481865 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.483781 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.484390 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.484401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.484781 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:02.485249 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.485262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.486121 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.486127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.487726 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.487741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.488245 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.488251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.488507 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.489199 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:13:02.489226 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:02.490878 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:02.492774 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:02.492779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:02.492795 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:02.493054 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:02.493356 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:02.493360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:02.493362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:02.493506 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:02.493988 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:02.493996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:02.494377 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.494610 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.494986 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260806 16:13:02.497395 29509 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36667
I20260806 16:13:02.497668 29509 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:02.498168 29509 heartbeater.cc:511] Master 127.18.90.62:36667 requested a full tablet report, sending...
I20260806 16:13:02.499287 29181 ts_manager.cc:194] Registered new tserver with Master: 7d5e5fc8eb0d4d52aaa5d0521c952ed7 (127.18.90.2:33853)
I20260806 16:13:02.500176 29181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:35877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:13:02.512100 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36667
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:02.624714 29514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:02.624982 29514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:02.625026 29514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:02.629158 29514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:02.629226 29514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:02.629287 29514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:13:02.634586 29514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:02.635615 29514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:02.636686 29514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:02.646569 29522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:02.646590 29520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:02.646569 29519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:02.646868 29514 server_base.cc:1034] running on GCE node
I20260806 16:13:02.647467 29514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:02.648124 29514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:02.649331 29514 hybrid_clock.cc:648] HybridClock initialized: now 1786032782649314 us; error 28 us; skew 500 ppm
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032782, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:02.652100 29514 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:13:02.653049 29514 webserver.cc:460] Webserver started at http://127.18.90.3:33617/ using document root <none> and password file <none>
I20260806 16:13:02.653635 29514 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:02.653740 29514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:02.653965 29514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:13:02.656061 29514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "5001e0d30f7d46b88680662f5c1a6faa"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "55bb5c3db8f672bd26e1a042d20a9688"
server_key_iv: "90c703dccf05b13720090f62c7994cd5"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.656733 29514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "5001e0d30f7d46b88680662f5c1a6faa"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "55bb5c3db8f672bd26e1a042d20a9688"
server_key_iv: "90c703dccf05b13720090f62c7994cd5"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.660542 29514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:13:02.662954 29529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.664142 29514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:13:02.664324 29514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "5001e0d30f7d46b88680662f5c1a6faa"
format_stamp: "Formatted at 2026-08-06 16:13:02 on dist-test-slave-0b58"
server_key: "55bb5c3db8f672bd26e1a042d20a9688"
server_key_iv: "90c703dccf05b13720090f62c7994cd5"
server_key_version: "encryptionkey@0"
I20260806 16:13:02.664444 29514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:02.678731 29514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:02.681682 29514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:02.681893 29514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:02.682574 29514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:02.683405 29514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:02.683475 29514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.683550 29514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:02.683594 29514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:02.692790 29514 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:41121
I20260806 16:13:02.692816 29643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:41121 every 8 connection(s)
I20260806 16:13:02.693702 29514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:13:02.699069 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 29514
I20260806 16:13:02.699184 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:13:02.699517 18792 external_mini_cluster.cc:1428] Setting key 7f91761792dc58970ccb8a68f820bca2
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032782, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:13:02.707545 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.695454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:41815 (local address 127.18.90.62:36667)
0806 16:13:02.695613 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.695618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.696535 (+ 917us) server_negotiation.cc:407] Connection header received
0806 16:13:02.697932 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.697935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.697984 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.698059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:02.699262 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.699983 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.701398 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.701670 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.701779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.704275 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:02.704282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:02.704286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:02.704317 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:02.705544 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.706045 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.706050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.706052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.706141 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.706491 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.706496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.706497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.706676 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:02.706782 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.707264 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.707382 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:13:02.708400 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.695861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36667 (local address 127.18.90.3:41815)
0806 16:13:02.696357 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.696426 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.697729 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.698219 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.698226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.698573 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:02.699070 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.699084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.700100 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.700103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.701277 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.701288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.701818 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.701822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.701964 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.702609 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:13:02.702625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:02.704094 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:02.705662 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:02.705675 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:02.705689 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:02.705941 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:02.706254 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:02.706258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:02.706260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:02.706397 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:02.706781 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:02.706788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:02.707032 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.707222 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.707533 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260806 16:13:02.709463 29644 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36667
I20260806 16:13:02.709715 29644 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:02.710201 29644 heartbeater.cc:511] Master 127.18.90.62:36667 requested a full tablet report, sending...
I20260806 16:13:02.711222 29181 ts_manager.cc:194] Registered new tserver with Master: 5001e0d30f7d46b88680662f5c1a6faa (127.18.90.3:41121)
I20260806 16:13:02.711745 29181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:41815
I20260806 16:13:02.713490 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:13:02.722332 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.715437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.18.90.62:36667)
0806 16:13:02.715571 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.715574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.715754 (+ 180us) server_negotiation.cc:407] Connection header received
0806 16:13:02.715841 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.715843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.715877 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.715940 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:02.716674 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.717164 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.717943 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.718146 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.718218 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.719455 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:02.719457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:02.719460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:02.719480 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:02.720922 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.721318 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.721321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.721322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.721392 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:02.721615 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:02.721617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:02.721618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:02.721815 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:02.721917 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.722107 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.722200 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:13:02.725349 29181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:13:02.728463 29181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:13:02.743527 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.738098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37145 (local address 127.0.0.1:36792)
0806 16:13:02.738851 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.738866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.738959 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.739514 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.739516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.739552 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:02.739662 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.739668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.740824 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.740829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.741981 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.741991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.742944 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.742948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.743063 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.743105 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.743219 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.743367 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":64,"thread_start_us":103,"threads_started":1}
I20260806 16:13:02.743597 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.738257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.18.90.1:37145)
0806 16:13:02.738707 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.738722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.738964 (+ 242us) server_negotiation.cc:407] Connection header received
0806 16:13:02.739050 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.739054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.739211 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.739346 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:02.739799 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.740665 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.742112 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.742814 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.742921 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.743059 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.743272 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.743377 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":106,"threads_started":1}
I20260806 16:13:02.744050 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.737316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:33853 (local address 127.0.0.1:52246)
0806 16:13:02.738510 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.738564 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.738705 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.739284 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.739291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.739313 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:02.739470 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.739477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.741038 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.741043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.742421 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.742430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.743567 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.743572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.743666 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.743691 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.743854 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.743935 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":63,"spinlock_wait_cycles":6528,"thread_start_us":89,"threads_started":1}
I20260806 16:13:02.744428 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.737320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41121 (local address 127.0.0.1:53648)
0806 16:13:02.738510 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.738564 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.738706 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.739617 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.739619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.739640 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:02.739747 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.739755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.741149 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.741152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.742419 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.742430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.743991 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.743995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.744086 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.744109 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.744188 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.744291 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":268,"spinlock_wait_cycles":4480,"thread_start_us":90,"threads_started":1}
I20260806 16:13:02.744531 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.737448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.18.90.2:33853)
0806 16:13:02.738157 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.738526 (+ 369us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.738668 (+ 142us) server_negotiation.cc:407] Connection header received
0806 16:13:02.738885 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.738889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.739113 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.739286 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:02.739590 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.740873 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.742572 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.743279 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.743375 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.743503 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.744233 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.744354 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":121,"threads_started":1}
I20260806 16:13:02.744997 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.737580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.18.90.3:41121)
0806 16:13:02.739000 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.739012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.739036 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:13:02.739108 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.739112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.739306 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.739451 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:02.739877 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.741021 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.742550 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.743373 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.743480 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.744671 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.744727 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.744829 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":142,"threads_started":1}
I20260806 16:13:02.746737 29308 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6fe20198843b9b896725bf36ce7e2 (DEFAULT_TABLE table=test-table [id=9724a9e7d8b84c58b47ed842d5b5769b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:13:02.747905 29308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6fe20198843b9b896725bf36ce7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:02.747587 29442 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6fe20198843b9b896725bf36ce7e2 (DEFAULT_TABLE table=test-table [id=9724a9e7d8b84c58b47ed842d5b5769b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:13:02.747659 29577 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6fe20198843b9b896725bf36ce7e2 (DEFAULT_TABLE table=test-table [id=9724a9e7d8b84c58b47ed842d5b5769b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:13:02.748530 29442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6fe20198843b9b896725bf36ce7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:02.748687 29577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6fe20198843b9b896725bf36ce7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:02.755227 29663 tablet_bootstrap.cc:492] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb: Bootstrap starting.
I20260806 16:13:02.755802 29661 tablet_bootstrap.cc:492] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Bootstrap starting.
I20260806 16:13:02.755925 29662 tablet_bootstrap.cc:492] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Bootstrap starting.
I20260806 16:13:02.758226 29662 tablet_bootstrap.cc:654] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:02.758729 29661 tablet_bootstrap.cc:654] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:02.759601 29661 log.cc:824] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Log is configured to *not* fsync() on all Append() calls
I20260806 16:13:02.759601 29662 log.cc:824] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:13:02.761487 29661 tablet_bootstrap.cc:492] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: No bootstrap required, opened a new log
I20260806 16:13:02.761713 29661 ts_tablet_manager.cc:1397] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:13:02.761818 29662 tablet_bootstrap.cc:492] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: No bootstrap required, opened a new log
I20260806 16:13:02.761994 29662 ts_tablet_manager.cc:1397] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:13:02.762156 29663 tablet_bootstrap.cc:654] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:02.763417 29663 log.cc:824] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb: Log is configured to *not* fsync() on all Append() calls
I20260806 16:13:02.764981 29661 raft_consensus.cc:348] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.765249 29661 raft_consensus.cc:374] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:02.765312 29661 raft_consensus.cc:729] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5e5fc8eb0d4d52aaa5d0521c952ed7, State: Initialized, Role: FOLLOWER
I20260806 16:13:02.765511 29663 tablet_bootstrap.cc:492] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb: No bootstrap required, opened a new log
I20260806 16:13:02.765668 29663 ts_tablet_manager.cc:1397] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260806 16:13:02.765571 29662 raft_consensus.cc:348] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.765779 29661 consensus_queue.cc:260] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.765872 29662 raft_consensus.cc:374] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:02.765929 29662 raft_consensus.cc:729] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5001e0d30f7d46b88680662f5c1a6faa, State: Initialized, Role: FOLLOWER
I20260806 16:13:02.766566 29509 heartbeater.cc:503] Master 127.18.90.62:36667 was elected leader, sending a full tablet report...
I20260806 16:13:02.767024 29661 ts_tablet_manager.cc:1428] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:13:02.766395 29662 consensus_queue.cc:260] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.768517 29662 ts_tablet_manager.cc:1428] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:13:02.768883 29644 heartbeater.cc:503] Master 127.18.90.62:36667 was elected leader, sending a full tablet report...
I20260806 16:13:02.769331 29663 raft_consensus.cc:348] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.769569 29663 raft_consensus.cc:374] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:02.769618 29663 raft_consensus.cc:729] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c91d4adc347473485c643d6f4cdb2cb, State: Initialized, Role: FOLLOWER
I20260806 16:13:02.770335 29663 consensus_queue.cc:260] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.771314 29663 ts_tablet_manager.cc:1428] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:13:02.771385 29374 heartbeater.cc:503] Master 127.18.90.62:36667 was elected leader, sending a full tablet report...
I20260806 16:13:02.834746 29669 raft_consensus.cc:482] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:13:02.834983 29669 raft_consensus.cc:504] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.836148 29669 leader_election.cc:288] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d5e5fc8eb0d4d52aaa5d0521c952ed7 (127.18.90.2:33853), 5001e0d30f7d46b88680662f5c1a6faa (127.18.90.3:41121)
I20260806 16:13:02.840132 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.836839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:60779 (local address 127.18.90.3:41121)
0806 16:13:02.836979 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.836983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.837020 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:13:02.837078 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.837080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.837131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.837206 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:02.837482 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.837987 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.839165 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.839564 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.839699 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.839767 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.839931 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.839998 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:13:02.840173 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.836519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:36569 (local address 127.18.90.2:33853)
0806 16:13:02.836652 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.836657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.836812 (+ 155us) server_negotiation.cc:407] Connection header received
0806 16:13:02.837034 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.837040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.837100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.837172 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:02.837592 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.838076 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.839187 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.839590 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.839715 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.839789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.839975 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.840047 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:13:02.840188 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.836369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:33853 (local address 127.18.90.1:36569)
0806 16:13:02.836725 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.836741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.836896 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.837336 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.837338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.837354 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:02.837445 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.837449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.838198 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.838203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.839047 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.839056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.839720 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.839723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.839845 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.839864 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.839955 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.840027 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":43,"thread_start_us":116,"threads_started":1}
I20260806 16:13:02.840212 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.836409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41121 (local address 127.18.90.1:60779)
0806 16:13:02.836880 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.836892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:02.836970 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:02.837235 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:02.837239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:02.837256 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:02.837372 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.837377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.838096 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.838103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.839046 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:02.839056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.839695 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.839699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:02.839795 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.839828 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 16:13:02.839915 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 16:13:02.839986 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":48,"thread_start_us":124,"threads_started":1}
I20260806 16:13:02.840803 29597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6fe20198843b9b896725bf36ce7e2" candidate_uuid: "9c91d4adc347473485c643d6f4cdb2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5001e0d30f7d46b88680662f5c1a6faa" is_pre_election: true
I20260806 16:13:02.840922 29462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6fe20198843b9b896725bf36ce7e2" candidate_uuid: "9c91d4adc347473485c643d6f4cdb2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" is_pre_election: true
I20260806 16:13:02.841141 29597 raft_consensus.cc:2454] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c91d4adc347473485c643d6f4cdb2cb in term 0.
I20260806 16:13:02.841217 29462 raft_consensus.cc:2454] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c91d4adc347473485c643d6f4cdb2cb in term 0.
I20260806 16:13:02.841635 29260 leader_election.cc:302] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5001e0d30f7d46b88680662f5c1a6faa, 9c91d4adc347473485c643d6f4cdb2cb; no voters:
I20260806 16:13:02.841923 29669 raft_consensus.cc:2781] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:13:02.842010 29669 raft_consensus.cc:482] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:13:02.842054 29669 raft_consensus.cc:3037] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:02.843353 29669 raft_consensus.cc:504] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.843796 29669 leader_election.cc:288] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [CANDIDATE]: Term 1 election: Requested vote from peers 7d5e5fc8eb0d4d52aaa5d0521c952ed7 (127.18.90.2:33853), 5001e0d30f7d46b88680662f5c1a6faa (127.18.90.3:41121)
I20260806 16:13:02.844141 29462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6fe20198843b9b896725bf36ce7e2" candidate_uuid: "9c91d4adc347473485c643d6f4cdb2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7"
I20260806 16:13:02.844321 29462 raft_consensus.cc:3037] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:02.844416 29597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6fe20198843b9b896725bf36ce7e2" candidate_uuid: "9c91d4adc347473485c643d6f4cdb2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5001e0d30f7d46b88680662f5c1a6faa"
I20260806 16:13:02.844529 29597 raft_consensus.cc:3037] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:02.845724 29462 raft_consensus.cc:2454] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c91d4adc347473485c643d6f4cdb2cb in term 1.
I20260806 16:13:02.845878 29597 raft_consensus.cc:2454] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c91d4adc347473485c643d6f4cdb2cb in term 1.
I20260806 16:13:02.846180 29262 leader_election.cc:302] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d5e5fc8eb0d4d52aaa5d0521c952ed7, 9c91d4adc347473485c643d6f4cdb2cb; no voters:
I20260806 16:13:02.846403 29669 raft_consensus.cc:2781] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:13:02.846676 29669 raft_consensus.cc:686] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [term 1 LEADER]: Becoming Leader. State: Replica: 9c91d4adc347473485c643d6f4cdb2cb, State: Running, Role: LEADER
I20260806 16:13:02.847025 29669 consensus_queue.cc:237] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } }
I20260806 16:13:02.850252 29181 catalog_manager.cc:5495] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c91d4adc347473485c643d6f4cdb2cb (127.18.90.1). New cstate: current_term: 1 leader_uuid: "9c91d4adc347473485c643d6f4cdb2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c91d4adc347473485c643d6f4cdb2cb" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37145 } health_report { overall_health: HEALTHY } } }
I20260806 16:13:02.869222 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.865404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.18.90.1:37145)
0806 16:13:02.865546 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:02.865550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:02.865613 (+ 63us) server_negotiation.cc:407] Connection header received
0806 16:13:02.865820 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:02.865823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:02.865879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:02.865987 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:13:02.866245 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.866755 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.867727 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:02.867946 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:02.868057 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:02.868403 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:13:02.868832 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:13:02.868967 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:02.869007 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 16:13:02.869070 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:13:02.875005 29597 raft_consensus.cc:1261] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 1 FOLLOWER]: Refusing update from remote peer 9c91d4adc347473485c643d6f4cdb2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:13:02.875005 29462 raft_consensus.cc:1261] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 1 FOLLOWER]: Refusing update from remote peer 9c91d4adc347473485c643d6f4cdb2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:13:02.875746 29669 consensus_queue.cc:1035] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d5e5fc8eb0d4d52aaa5d0521c952ed7" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:13:02.875918 29672 consensus_queue.cc:1035] T b9d6fe20198843b9b896725bf36ce7e2 P 9c91d4adc347473485c643d6f4cdb2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5001e0d30f7d46b88680662f5c1a6faa" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:13:02.884516 29677 mvcc.cc:205] Tried to move back new op lower bound from 7315590278649081856 to 7315590278544306176. Current Snapshot: MvccSnapshot[applied={T|T < 7315590278649081856}]
I20260806 16:13:02.885720 29676 mvcc.cc:205] Tried to move back new op lower bound from 7315590278649081856 to 7315590278544306176. Current Snapshot: MvccSnapshot[applied={T|T < 7315590278649081856}]
I20260806 16:13:02.893568 29182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:13:02.893766 29182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:13:02.896379 29182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fef8c3d4f8ba413a90d4b399dc013719: Sending DeleteTablet for 3 replicas of tablet b9d6fe20198843b9b896725bf36ce7e2
I20260806 16:13:02.897156 29442 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6fe20198843b9b896725bf36ce7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:13:02 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:52246
I20260806 16:13:02.897289 29308 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6fe20198843b9b896725bf36ce7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:13:02 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:36792
I20260806 16:13:02.897807 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 29244
I20260806 16:13:02.897936 29577 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6fe20198843b9b896725bf36ce7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:13:02 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:53648
I20260806 16:13:02.898259 29684 tablet_replica.cc:330] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: stopping tablet replica
I20260806 16:13:02.898794 29684 raft_consensus.cc:2229] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:13:02.899145 29684 raft_consensus.cc:2258] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:13:02.900646 29684 ts_tablet_manager.cc:1905] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:13:02.903090 29684 ts_tablet_manager.cc:1918] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:13:02.903175 29684 log.cc:1196] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/b9d6fe20198843b9b896725bf36ce7e2
I20260806 16:13:02.903481 29684 ts_tablet_manager.cc:1939] T b9d6fe20198843b9b896725bf36ce7e2 P 7d5e5fc8eb0d4d52aaa5d0521c952ed7: Deleting consensus metadata
I20260806 16:13:02.904800 29168 catalog_manager.cc:4841] TS 7d5e5fc8eb0d4d52aaa5d0521c952ed7 (127.18.90.2:33853): tablet b9d6fe20198843b9b896725bf36ce7e2 (table test-table [id=9724a9e7d8b84c58b47ed842d5b5769b]) successfully deleted
I20260806 16:13:02.905207 29686 tablet_replica.cc:330] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: stopping tablet replica
I20260806 16:13:02.905565 29686 raft_consensus.cc:2229] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:13:02.905912 29686 raft_consensus.cc:2258] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 16:13:02.907225 29166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:37145 (error 108)
I20260806 16:13:02.907492 29686 ts_tablet_manager.cc:1905] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:13:02.908111 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 29379
I20260806 16:13:02.908350 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:02.908012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37145 (local address 127.0.0.1:36820)
0806 16:13:02.908160 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:02.908231 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:37145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 16:13:02.909057 29166 catalog_manager.cc:4584] TS 9c91d4adc347473485c643d6f4cdb2cb (127.18.90.1:37145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9d6fe20198843b9b896725bf36ce7e2: Network error: Client connection negotiation failed: client connection to 127.18.90.1:37145: connect: Connection refused (error 111)
I20260806 16:13:02.910181 29686 ts_tablet_manager.cc:1918] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:13:02.910269 29686 log.cc:1196] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/b9d6fe20198843b9b896725bf36ce7e2
I20260806 16:13:02.910598 29686 ts_tablet_manager.cc:1939] T b9d6fe20198843b9b896725bf36ce7e2 P 5001e0d30f7d46b88680662f5c1a6faa: Deleting consensus metadata
I20260806 16:13:02.911697 29166 catalog_manager.cc:4841] TS 5001e0d30f7d46b88680662f5c1a6faa (127.18.90.3:41121): tablet b9d6fe20198843b9b896725bf36ce7e2 (table test-table [id=9724a9e7d8b84c58b47ed842d5b5769b]) successfully deleted
I20260806 16:13:02.915661 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 29514
I20260806 16:13:02.921913 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 29150
2026-08-06T16:13:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2321 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29690](info): set up 2 sockets
Aug 06 16:13:02 dist-test-slave-0b58 krb5kdc[29690](info): commencing operation
krb5kdc: starting...
W20260806 16:13:03.969843 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:13:04 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:13:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/rangeradmin_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/rangeradmin_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/rangerlookup_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/rangerlookup_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
I20260806 16:13:04.146859 18792 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskG62Umc/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/postgres -l logfile start
2026-08-06 16:13:06.765 UTC [29720] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 16:13:06.765 UTC [29720] LOG: listening on IPv4 address "127.18.90.20", port 49801
2026-08-06 16:13:06.768 UTC [29720] LOG: listening on Unix socket "/tmp/.s.PGSQL.49801"
2026-08-06 16:13:06.790 UTC [29723] LOG: database system was shut down at 2026-08-06 16:13:05 UTC
2026-08-06 16:13:06.794 UTC [29720] LOG: database system is ready to accept connections
I20260806 16:13:07.749500 18792 mini_postgres.cc:96] Postgres bound to 49801
2026-08-06 16:13:07.758 UTC [29731] FATAL: database "slave" does not exist
127.18.90.20:49801 - accepting connections
I20260806 16:13:07.759114 18792 mini_ranger.cc:161] Starting Ranger...
I20260806 16:13:07.775828 18792 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 16:13:08.242722 18792 mini_ranger.cc:87] Created ranger Postgres database
I20260806 16:13:08.242812 18792 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-admin
2026-08-06 16:13:08,553 [I] DB FLAVOR :POSTGRES
2026-08-06 16:13:08,554 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:13:08,554 [I] Checking connection..
2026-08-06 16:13:08,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 16:13:08,832 [I] Checking connection passed.
2026-08-06 16:13:08,832 [I] --------- Verifying version history table ---------
2026-08-06 16:13:08,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:13:09,087 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:13:09,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:13:09,356 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:13:09,356 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 16:13:09,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 16:13:09.600 UTC [29796] WARNING: there is no transaction in progress
2026-08-06 16:13:09,607 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 16:13:09,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:13:09,851 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 16:13:09,851 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:13:09,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 16:13:10,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 16:13:10,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0b58.c.gcp-upstream.internal','N') ;"
2026-08-06 16:13:10,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:13:10,854 [I] Table x_portal_user does not exist in database ranger
2026-08-06 16:13:10,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:13:11,097 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 16:13:11,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:13:11,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:13:11,605 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 16:13:11,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 16:13:12.589 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.606 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.623 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.639 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.656 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.677 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.731 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.739 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.752 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.763 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.772 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.787 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.797 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.804 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.811 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:12.817 UTC [29904] WARNING: there is no transaction in progress
2026-08-06 16:13:13,489 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 16:13:13,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 16:13:13,750 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 16:13:13,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:13:13,984 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 16:13:13,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:13:14,243 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 16:13:14,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:13:14,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:13:14,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0b58.c.gcp-upstream.internal';"
2026-08-06 16:13:14,975 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 16:13:14,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 16:13:15,207 [I] DB_PATCHES have already been applied
I20260806 16:13:15.213594 18792 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-admin:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskG62Umc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 16:13:15.213687 18792 mini_ranger.cc:193] Using host: 127.18.90.20
I20260806 16:13:15.218041 18792 mini_ranger.cc:227] Ranger admin URL: http://127.18.90.20:51073
Aug 06, 2026 4:13:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:13:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:13:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:13:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:13:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:13:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:13:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 16:13:16.355593 18792 mini_ranger.cc:160] Time spent starting Ranger: real 8.596s user 0.000s sys 0.012s
Aug 06, 2026 4:13:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:13:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 06, 2026 4:13:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:13:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:13:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 16:13:37.810544 18792 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/rangerkms_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/rangerkms_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.90.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.90.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 16:13:37 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032817, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 16:13:37.910948 18792 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 16:13:37.922024 18792 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 16:13:38.790612 18792 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 16:13:38.790720 18792 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms
I20260806 16:13:38.790832 18792 mini_ranger_kms.cc:227] Using postgres at 127.18.90.20:49801
2026-08-06 16:13:39,012 [I] DB FLAVOR :POSTGRES
2026-08-06 16:13:39,013 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:13:39,013 [I] Checking connection
2026-08-06 16:13:39,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 16:13:39,251 [I] connection success
2026-08-06 16:13:39,252 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 16:13:39,252 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 16:13:39,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 16:13:39,522 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 16:13:39,523 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:13:39,523 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 16:13:39,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:49801/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 16:13:39,816 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 16:13:40.293588 18792 mini_ranger_kms.cc:310] Created kms service
I20260806 16:13:40.506111 18792 mini_ranger_kms.cc:326] Created kudu user
I20260806 16:13:40.555294 18792 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 16:13:40 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032817, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:13:40.944973 18792 mini_ranger_kms.cc:384] Added ranger policy
I20260806 16:13:40.945147 18792 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms:/tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 16:13:40.945173 18792 mini_ranger_kms.cc:240] Using host: 127.18.90.20
I20260806 16:13:40.949712 18792 mini_ranger_kms.cc:276] Ranger KMS PID: 30084
I20260806 16:13:40.949872 18792 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.90.20:55947
Aug 06, 2026 4:13:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:13:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 16:13:43.160347 18792 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.249s user 0.037s sys 0.170s
I20260806 16:13:43.160490 18792 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 16:13:43.160631 18792 mini_ranger_kms.cc:401] 127.18.90.20:55947/kms/v1/keys
Aug 06 16:13:43 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.90.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 16:13:43 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.90.20@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
Aug 06 16:13:44 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.90.20@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
Aug 06 16:13:46 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:13:46.456470 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:43305
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:43305
--ranger_config_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:55947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:46.573223 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:46.573529 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:46.573568 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:46.577984 30136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:13:46.578068 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:46.578090 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:46.578107 30136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:13:46.578148 30136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:13:46.584074 30136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.90.20:55947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:43305
--ranger_config_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:46.585307 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:46.586444 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:46.595263 30142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:46.595265 30141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:46.595599 30136 server_base.cc:1034] running on GCE node
W20260806 16:13:46.595263 30144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:46.596417 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:46.597680 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:46.598865 30136 hybrid_clock.cc:648] HybridClock initialized: now 1786032826598845 us; error 38 us; skew 500 ppm
Aug 06 16:13:46 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032826, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:46.607303 30136 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:13:46.608536 30136 webserver.cc:460] Webserver started at http://127.18.90.62:45563/ using document root <none> and password file <none>
I20260806 16:13:46.609128 30136 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:46.609185 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:46.609436 30136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:13:46 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032826, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:13:46.675303 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "efc84f0c661a4efd9ed17de471e6a0a1"
format_stamp: "Formatted at 2026-08-06 16:13:46 on dist-test-slave-0b58"
server_key: "5cd56a37693454e7d130aa7597da8946"
server_key_iv: "ab0ffc9da0f403fa7eabfc4a8fc308e3"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:46.675995 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "efc84f0c661a4efd9ed17de471e6a0a1"
format_stamp: "Formatted at 2026-08-06 16:13:46 on dist-test-slave-0b58"
server_key: "5cd56a37693454e7d130aa7597da8946"
server_key_iv: "ab0ffc9da0f403fa7eabfc4a8fc308e3"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:46.680209 30136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:13:46.698254 30151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:46.699995 30136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:13:46.700158 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "efc84f0c661a4efd9ed17de471e6a0a1"
format_stamp: "Formatted at 2026-08-06 16:13:46 on dist-test-slave-0b58"
server_key: "5cd56a37693454e7d130aa7597da8946"
server_key_iv: "ab0ffc9da0f403fa7eabfc4a8fc308e3"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:46.700351 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:46.747584 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:46.751047 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:46.751259 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:46.760403 30136 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:43305
I20260806 16:13:46.760437 30203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:43305 every 8 connection(s)
I20260806 16:13:46.761699 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:13:46.764766 30204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:46.766351 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 30136
I20260806 16:13:46.766497 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:13:46.766801 18792 external_mini_cluster.cc:1428] Setting key 76ff401d431e7ecdfb1a805fbdf0a36c
I20260806 16:13:46.771759 30204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1: Bootstrap starting.
I20260806 16:13:46.774390 30204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:46.775329 30204 log.cc:824] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:13:46 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:13:46.777758 30204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1: No bootstrap required, opened a new log
I20260806 16:13:46.782018 30204 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43305 } }
I20260806 16:13:46.782320 30204 raft_consensus.cc:374] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:46.782404 30204 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc84f0c661a4efd9ed17de471e6a0a1, State: Initialized, Role: FOLLOWER
I20260806 16:13:46.782932 30204 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43305 } }
I20260806 16:13:46.783116 30204 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:13:46.783196 30204 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:13:46.783322 30204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:46.784389 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:46.768360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.18.90.62:43305)
0806 16:13:46.768858 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:46.768880 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:46.768926 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:13:46.769835 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:46.769862 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:46.770318 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:46.770694 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:46.771548 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:46.773050 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:46.774280 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:46.775082 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:46.775253 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:46.778378 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:46.778385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:46.778402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:46.778444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:46.781056 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:46.781567 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:46.781573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:46.781580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:46.781706 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:46.782005 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:46.782009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:46.782011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:46.782564 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:46.782748 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:46.783061 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 16:13:46.783389 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":148,"threads_started":1}
I20260806 16:13:46.784516 30204 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43305 } }
I20260806 16:13:46.784909 30204 leader_election.cc:302] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc84f0c661a4efd9ed17de471e6a0a1; no voters:
I20260806 16:13:46.785257 30204 leader_election.cc:288] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:13:46.785425 30209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:13:46.785686 30209 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [term 1 LEADER]: Becoming Leader. State: Replica: efc84f0c661a4efd9ed17de471e6a0a1, State: Running, Role: LEADER
I20260806 16:13:46.786041 30209 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43305 } }
I20260806 16:13:46.786605 30204 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:13:46.787374 30209 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43305 } } }
I20260806 16:13:46.787560 30209 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 16:13:46.787786 30209 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc84f0c661a4efd9ed17de471e6a0a1. Latest consensus state: current_term: 1 leader_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc84f0c661a4efd9ed17de471e6a0a1" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43305 } } }
I20260806 16:13:46.787868 30209 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 16:13:46.788143 30217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:13:46.792495 30217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:13:46.793028 30204 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 16:13:46.803398 30217 catalog_manager.cc:1347] Generated new cluster ID: 208ee2367bc54af9b8df89d70566fa84
I20260806 16:13:46.803550 30217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:13:46.825713 30217 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:13:46.827186 30217 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:13:46.834968 30217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1: Generated new TSK 0
I20260806 16:13:46.836921 30217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:13:47.627118 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:55947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:43305
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:47.756486 30249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:47.756772 30249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:47.756848 30249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:47.761397 30249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:47.761543 30249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:47.761672 30249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:13:47.769423 30249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.90.20:55947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:47.770872 30249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:47.772150 30249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:47.784327 30256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:47.784334 30259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:47.784389 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:47.785149 30249 server_base.cc:1034] running on GCE node
I20260806 16:13:47.785612 30249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:47.786307 30249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:47.787497 30249 hybrid_clock.cc:648] HybridClock initialized: now 1786032827787454 us; error 58 us; skew 500 ppm
Aug 06 16:13:47 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032827, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:47.791265 30249 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:13:47.792376 30249 webserver.cc:460] Webserver started at http://127.18.90.1:35877/ using document root <none> and password file <none>
I20260806 16:13:47.792997 30249 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:47.793146 30249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:47.793402 30249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:13:47 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032827, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:13:47.808363 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "c5ba6aa4559c45fdb20a077d1349c752"
format_stamp: "Formatted at 2026-08-06 16:13:47 on dist-test-slave-0b58"
server_key: "7039a96e501bae5e48030f4b483abf00"
server_key_iv: "375e93766ec11262025fac736c0f69c1"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:47.809310 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "c5ba6aa4559c45fdb20a077d1349c752"
format_stamp: "Formatted at 2026-08-06 16:13:47 on dist-test-slave-0b58"
server_key: "7039a96e501bae5e48030f4b483abf00"
server_key_iv: "375e93766ec11262025fac736c0f69c1"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:47.814229 30249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:13:47.835994 30266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:47.837683 30249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:13:47.837841 30249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "c5ba6aa4559c45fdb20a077d1349c752"
format_stamp: "Formatted at 2026-08-06 16:13:47 on dist-test-slave-0b58"
server_key: "7039a96e501bae5e48030f4b483abf00"
server_key_iv: "375e93766ec11262025fac736c0f69c1"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:47.837996 30249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:47.851189 30249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:47.855141 30249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:47.855415 30249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:47.856220 30249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:47.857396 30249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:47.857513 30249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:47.857650 30249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:47.857744 30249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:47.870347 30249 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:43941
I20260806 16:13:47.871661 30249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:13:47.878975 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 30249
I20260806 16:13:47.879110 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:13:47.879431 18792 external_mini_cluster.cc:1428] Setting key 5a1383447a318474622925616210952a
I20260806 16:13:47.885020 30381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:43941 every 8 connection(s)
Aug 06 16:13:47 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032827, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:13:47.912480 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:47.888582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:55219 (local address 127.18.90.62:43305)
0806 16:13:47.889022 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:47.889027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:47.898023 (+ 8996us) server_negotiation.cc:407] Connection header received
0806 16:13:47.899421 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:47.899428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:47.899508 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:47.899623 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:47.901697 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:47.902251 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:47.903457 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:47.903899 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:47.904027 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:47.907742 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:47.907747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:47.907750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:47.907779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:47.909862 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:47.910527 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:47.910530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:47.910532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:47.910627 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:47.911057 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:47.911061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:47.911063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:47.911292 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:47.911375 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:47.912112 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 16:13:47.912282 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":84,"threads_started":1}
I20260806 16:13:47.914445 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:47.887395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:43305 (local address 127.18.90.1:55219)
0806 16:13:47.897805 (+ 10410us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:47.897887 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:47.899105 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:47.900621 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:47.900636 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:47.900997 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:47.901526 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:47.901541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:47.902394 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:47.902398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:47.903303 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:47.903322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:47.904288 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:47.904295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:47.904554 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:47.905412 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:13:47.905440 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:47.907493 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:47.910009 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:47.910015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:47.910034 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:47.910400 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:47.910738 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:47.910741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:47.910743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:47.910944 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:47.911441 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:47.911448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:47.911796 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 16:13:47.912694 (+ 898us) client_negotiation.cc:240] Negotiation successful
0806 16:13:47.913099 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10100,"thread_start_us":181,"threads_started":1}
I20260806 16:13:47.916160 30382 heartbeater.cc:347] Connected to a master server at 127.18.90.62:43305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:13:47.916440 30382 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:47.917081 30382 heartbeater.cc:511] Master 127.18.90.62:43305 requested a full tablet report, sending...
I20260806 16:13:47.918956 30168 ts_manager.cc:194] Registered new tserver with Master: c5ba6aa4559c45fdb20a077d1349c752 (127.18.90.1:43941)
I20260806 16:13:47.920581 30168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:55219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:13:47.937896 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:55947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:43305
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:13:47 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032827, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 16:13:48.069898 30390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:48.070187 30390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:48.070304 30390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:48.074489 30390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:48.074612 30390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:48.074736 30390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:13:48.080346 30390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.90.20:55947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:48.081426 30390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:48.082526 30390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:48.093694 30396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:48.093653 30399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:48.094116 30397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:48.094636 30390 server_base.cc:1034] running on GCE node
I20260806 16:13:48.095141 30390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:48.095854 30390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:48.097035 30390 hybrid_clock.cc:648] HybridClock initialized: now 1786032828096955 us; error 101 us; skew 500 ppm
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:48.099898 30390 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:13:48.100925 30390 webserver.cc:460] Webserver started at http://127.18.90.2:41387/ using document root <none> and password file <none>
I20260806 16:13:48.101507 30390 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:48.101603 30390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:48.101826 30390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:13:48.120554 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "0ed59d318db34469bee82c6a42c25da8"
format_stamp: "Formatted at 2026-08-06 16:13:48 on dist-test-slave-0b58"
server_key: "25975a733da95dfd8db29ac515065bd1"
server_key_iv: "b9ac567a486121e81634fad990a60251"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:48.121318 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "0ed59d318db34469bee82c6a42c25da8"
format_stamp: "Formatted at 2026-08-06 16:13:48 on dist-test-slave-0b58"
server_key: "25975a733da95dfd8db29ac515065bd1"
server_key_iv: "b9ac567a486121e81634fad990a60251"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:48.126444 30390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 16:13:48.150130 30406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:48.151978 30390 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260806 16:13:48.152118 30390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "0ed59d318db34469bee82c6a42c25da8"
format_stamp: "Formatted at 2026-08-06 16:13:48 on dist-test-slave-0b58"
server_key: "25975a733da95dfd8db29ac515065bd1"
server_key_iv: "b9ac567a486121e81634fad990a60251"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:48.152405 30390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:48.173419 30390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:48.177604 30390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:48.177908 30390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:48.178707 30390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:48.179872 30390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:48.179994 30390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:48.180114 30390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:48.180184 30390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:48.191321 30390 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:43103
I20260806 16:13:48.192668 30390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:13:48.197624 30520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:43103 every 8 connection(s)
I20260806 16:13:48.199373 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 30390
I20260806 16:13:48.199506 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:13:48.199844 18792 external_mini_cluster.cc:1428] Setting key 0fbd7059178377d7a798b0ef3f2c71fb
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:13:48.230988 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.202027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:33113 (local address 127.18.90.62:43305)
0806 16:13:48.202341 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.202346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.206276 (+ 3930us) server_negotiation.cc:407] Connection header received
0806 16:13:48.208919 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.208923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.208996 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.209061 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:48.213714 (+ 4653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.214564 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.221720 (+ 7156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.222116 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.222220 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.226061 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:48.226065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:48.226069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:48.226105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:48.228120 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:48.228896 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:48.228900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:48.228903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:48.229002 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:48.229460 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:48.229463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:48.229465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:48.229705 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:48.229808 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.230606 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.230750 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260806 16:13:48.233178 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.200512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:43305 (local address 127.18.90.2:33113)
0806 16:13:48.206072 (+ 5560us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.206165 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.207436 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.209558 (+ 2122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.209578 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.210000 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:48.213463 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.213485 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.214757 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.214761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.216339 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.216351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.222505 (+ 6154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.222510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.222824 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.223657 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:13:48.223688 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:48.225707 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:48.228320 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:48.228327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:48.228348 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:48.228753 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:48.229121 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:48.229124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:48.229126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:48.229337 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:48.229886 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:48.229893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:48.230286 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.231538 (+ 1252us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.231953 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5293,"thread_start_us":153,"threads_started":1}
I20260806 16:13:48.235040 30521 heartbeater.cc:347] Connected to a master server at 127.18.90.62:43305
I20260806 16:13:48.235313 30521 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:13:48.235972 30521 heartbeater.cc:511] Master 127.18.90.62:43305 requested a full tablet report, sending...
I20260806 16:13:48.237303 30168 ts_manager.cc:194] Registered new tserver with Master: 0ed59d318db34469bee82c6a42c25da8 (127.18.90.2:43103)
I20260806 16:13:48.238121 30168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:33113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:13:48.256400 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:55947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:43305
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:13:48.423179 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:13:48.424003 30526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:13:48.424154 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:13:48.429857 30526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:13:48.430058 30526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:13:48.430202 30526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:13:48.437868 30526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.90.20:55947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:13:48.439491 30526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:13:48.441282 30526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:13:48.454396 30531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:13:48.455015 30532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:48.455130 30526 server_base.cc:1034] running on GCE node
W20260806 16:13:48.455328 30534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:13:48.456043 30526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:13:48.456838 30526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:13:48.458057 30526 hybrid_clock.cc:648] HybridClock initialized: now 1786032828457999 us; error 87 us; skew 500 ppm
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:13:48.461805 30526 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:13:48.462942 30526 webserver.cc:460] Webserver started at http://127.18.90.3:42159/ using document root <none> and password file <none>
I20260806 16:13:48.463691 30526 fs_manager.cc:359] Metadata directory not provided
I20260806 16:13:48.463835 30526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:13:48.464152 30526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:13:48.479131 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "dfd06800dc95482fa29d996726f5da64"
format_stamp: "Formatted at 2026-08-06 16:13:48 on dist-test-slave-0b58"
server_key: "4bdcd2d86993debbaad30f71a0e04471"
server_key_iv: "888a7dcf006c927916cfcd1ee6427ff8"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:48.479879 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "dfd06800dc95482fa29d996726f5da64"
format_stamp: "Formatted at 2026-08-06 16:13:48 on dist-test-slave-0b58"
server_key: "4bdcd2d86993debbaad30f71a0e04471"
server_key_iv: "888a7dcf006c927916cfcd1ee6427ff8"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:48.485699 30526 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 16:13:48.502947 30541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:48.504777 30526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 16:13:48.504994 30526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "dfd06800dc95482fa29d996726f5da64"
format_stamp: "Formatted at 2026-08-06 16:13:48 on dist-test-slave-0b58"
server_key: "4bdcd2d86993debbaad30f71a0e04471"
server_key_iv: "888a7dcf006c927916cfcd1ee6427ff8"
server_key_version: "kuduclusterkey@0"
I20260806 16:13:48.505225 30526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:13:48.528158 30526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:13:48.531973 30526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:13:48.532280 30526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:13:48.533022 30526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:13:48.534245 30526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:13:48.534364 30526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:48.534475 30526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:13:48.534538 30526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:13:48.545293 30526 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:46657
I20260806 16:13:48.547650 30526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:13:48.557361 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 30526
I20260806 16:13:48.557504 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:13:48.557801 18792 external_mini_cluster.cc:1428] Setting key 61f6f8f243b9f49180f9255b8aca6e5b
I20260806 16:13:48.558409 30656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:46657 every 8 connection(s)
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032827, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
Aug 06 16:13:48 dist-test-slave-0b58 krb5kdc[29690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:13:48.602698 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.562595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:37971 (local address 127.18.90.62:43305)
0806 16:13:48.562747 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.562751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.568877 (+ 6126us) server_negotiation.cc:407] Connection header received
0806 16:13:48.570275 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.570283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.570630 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.570722 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:48.580420 (+ 9698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.581262 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.584837 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.585165 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.585239 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.588730 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:48.588734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:48.588737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:48.588764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:48.592086 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:48.600776 (+ 8690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:48.600781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:48.600795 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:48.600901 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:48.601333 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:48.601337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:48.601338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:48.601571 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:48.601649 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.602402 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.602539 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:13:48.604744 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.561406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:43305 (local address 127.18.90.3:37971)
0806 16:13:48.568660 (+ 7254us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.568741 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.569950 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.573000 (+ 3050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.573010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.573404 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:13:48.574136 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.574156 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.581469 (+ 7313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.581474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.584682 (+ 3208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.584697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.585474 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.585477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.585698 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.586449 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:13:48.586471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:13:48.588530 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:13:48.600117 (+ 11587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:48.600126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:48.600150 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:48.600641 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:48.601015 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:13:48.601018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:13:48.601020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:13:48.601229 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:13:48.601718 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:13:48.601726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:13:48.602115 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.603180 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.603570 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6948,"thread_start_us":181,"threads_started":1}
I20260806 16:13:48.606431 30657 heartbeater.cc:347] Connected to a master server at 127.18.90.62:43305
I20260806 16:13:48.606675 30657 heartbeater.cc:464] Registering TS with master...
I20260806 16:13:48.607290 30657 heartbeater.cc:511] Master 127.18.90.62:43305 requested a full tablet report, sending...
I20260806 16:13:48.609057 30168 ts_manager.cc:194] Registered new tserver with Master: dfd06800dc95482fa29d996726f5da64 (127.18.90.3:46657)
I20260806 16:13:48.609751 30168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:37971
I20260806 16:13:48.611321 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:13:48.629235 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.615515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.18.90.62:43305)
0806 16:13:48.615638 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.615643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.619065 (+ 3422us) server_negotiation.cc:407] Connection header received
0806 16:13:48.619912 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.619915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.619973 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.620027 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:13:48.621300 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.622102 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.623352 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.623664 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.623732 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.625616 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:13:48.625620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:13:48.625623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:13:48.625652 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:13:48.627340 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:48.627842 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:48.627846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:48.627848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:48.627937 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:13:48.628234 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:13:48.628237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:13:48.628239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:13:48.628478 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:13:48.628590 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.628949 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.629075 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:13:48.633160 30168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:13:48.637571 30168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:13:48.663436 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.653213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:46657 (local address 127.0.0.1:37940)
0806 16:13:48.655922 (+ 2709us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.655932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.656000 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.657470 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.657476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.657500 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:48.657630 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.657636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.659248 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.659251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.661561 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.661569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.662889 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.662893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.663000 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.663042 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.663152 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.663273 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2661,"thread_start_us":79,"threads_started":1}
I20260806 16:13:48.664909 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.653615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.18.90.3:46657)
0806 16:13:48.655089 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.655101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.656084 (+ 983us) server_negotiation.cc:407] Connection header received
0806 16:13:48.656168 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.656174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.656422 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.656567 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:48.657753 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.659119 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.661711 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.662557 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.662656 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.662828 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.664604 (+ 1776us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.664724 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361,"thread_start_us":133,"threads_started":1}
I20260806 16:13:48.665501 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.650050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43941 (local address 127.0.0.1:41520)
0806 16:13:48.655293 (+ 5243us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.655349 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.655476 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.661076 (+ 5600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.661080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.661100 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:48.661207 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.661213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.662654 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.662657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.663739 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.663755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.665064 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.665067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.665168 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.665197 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.665269 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.665362 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4784,"thread_start_us":108,"threads_started":1}
I20260806 16:13:48.666277 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.651056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.18.90.1:43941)
0806 16:13:48.660479 (+ 9423us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.660485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.660509 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:13:48.660590 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.660596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.660788 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.660938 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:48.661357 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.662507 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.663888 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.664730 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.664841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.664991 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.666010 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.666112 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9298,"thread_start_us":107,"threads_started":1}
I20260806 16:13:48.668735 30590 tablet_service.cc:1467] Processing CreateTablet for tablet b88ad95ede064812b576d451694e69b1 (DEFAULT_TABLE table=test-table [id=f7884d51b42c4226b954ca4f03ed1fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:13:48.671664 30590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ad95ede064812b576d451694e69b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:48.672319 30316 tablet_service.cc:1467] Processing CreateTablet for tablet b88ad95ede064812b576d451694e69b1 (DEFAULT_TABLE table=test-table [id=f7884d51b42c4226b954ca4f03ed1fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:13:48.673417 30316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ad95ede064812b576d451694e69b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:48.676849 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.652583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:43103 (local address 127.0.0.1:45116)
0806 16:13:48.655688 (+ 3105us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.655699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.655762 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.660109 (+ 4347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.660112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.660131 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:48.660229 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.660236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.669786 (+ 9550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.669790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.670928 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.670940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.676385 (+ 5445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.676390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.676497 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.676529 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.676587 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.676701 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3049,"thread_start_us":85,"threads_started":1}
I20260806 16:13:48.677524 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.652766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.18.90.2:43103)
0806 16:13:48.659410 (+ 6644us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.659426 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.659453 (+ 27us) server_negotiation.cc:407] Connection header received
0806 16:13:48.659531 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.659536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.659778 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.659952 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:48.668362 (+ 8410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.669599 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.671118 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.671889 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.677068 (+ 5179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.677232 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.677287 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.677384 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6533,"thread_start_us":87,"threads_started":1}
I20260806 16:13:48.680505 30455 tablet_service.cc:1467] Processing CreateTablet for tablet b88ad95ede064812b576d451694e69b1 (DEFAULT_TABLE table=test-table [id=f7884d51b42c4226b954ca4f03ed1fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:13:48.681550 30455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ad95ede064812b576d451694e69b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:13:48.682433 30676 tablet_bootstrap.cc:492] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Bootstrap starting.
I20260806 16:13:48.686252 30676 tablet_bootstrap.cc:654] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:48.687238 30676 log.cc:824] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Log is configured to *not* fsync() on all Append() calls
I20260806 16:13:48.689571 30676 tablet_bootstrap.cc:492] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: No bootstrap required, opened a new log
I20260806 16:13:48.689765 30676 ts_tablet_manager.cc:1397] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260806 16:13:48.695763 30676 raft_consensus.cc:348] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.696230 30676 raft_consensus.cc:374] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:48.696348 30676 raft_consensus.cc:729] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd06800dc95482fa29d996726f5da64, State: Initialized, Role: FOLLOWER
I20260806 16:13:48.696920 30676 consensus_queue.cc:260] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.698148 30676 ts_tablet_manager.cc:1428] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20260806 16:13:48.698558 30657 heartbeater.cc:503] Master 127.18.90.62:43305 was elected leader, sending a full tablet report...
I20260806 16:13:48.704932 30678 tablet_bootstrap.cc:492] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8: Bootstrap starting.
I20260806 16:13:48.706969 30679 tablet_bootstrap.cc:492] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752: Bootstrap starting.
I20260806 16:13:48.708863 30678 tablet_bootstrap.cc:654] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:48.712373 30679 tablet_bootstrap.cc:654] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752: Neither blocks nor log segments found. Creating new log.
I20260806 16:13:48.712754 30678 log.cc:824] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:13:48.713675 30679 log.cc:824] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752: Log is configured to *not* fsync() on all Append() calls
I20260806 16:13:48.717708 30678 tablet_bootstrap.cc:492] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8: No bootstrap required, opened a new log
I20260806 16:13:48.717990 30678 ts_tablet_manager.cc:1397] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260806 16:13:48.722106 30678 raft_consensus.cc:348] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.722483 30678 raft_consensus.cc:374] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:48.722586 30678 raft_consensus.cc:729] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed59d318db34469bee82c6a42c25da8, State: Initialized, Role: FOLLOWER
I20260806 16:13:48.723824 30678 consensus_queue.cc:260] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.724997 30678 ts_tablet_manager.cc:1428] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 16:13:48.725168 30679 tablet_bootstrap.cc:492] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752: No bootstrap required, opened a new log
I20260806 16:13:48.725410 30521 heartbeater.cc:503] Master 127.18.90.62:43305 was elected leader, sending a full tablet report...
I20260806 16:13:48.725479 30679 ts_tablet_manager.cc:1397] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20260806 16:13:48.730293 30679 raft_consensus.cc:348] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.730665 30679 raft_consensus.cc:374] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:13:48.730764 30679 raft_consensus.cc:729] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ba6aa4559c45fdb20a077d1349c752, State: Initialized, Role: FOLLOWER
I20260806 16:13:48.731330 30679 consensus_queue.cc:260] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.732146 30382 heartbeater.cc:503] Master 127.18.90.62:43305 was elected leader, sending a full tablet report...
I20260806 16:13:48.732583 30679 ts_tablet_manager.cc:1428] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 16:13:48.762889 30684 raft_consensus.cc:482] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:13:48.763182 30684 raft_consensus.cc:504] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.764559 30684 leader_election.cc:288] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed59d318db34469bee82c6a42c25da8 (127.18.90.2:43103), dfd06800dc95482fa29d996726f5da64 (127.18.90.3:46657)
I20260806 16:13:48.770215 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.765458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:43907 (local address 127.18.90.3:46657)
0806 16:13:48.765589 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.765595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.765663 (+ 68us) server_negotiation.cc:407] Connection header received
0806 16:13:48.765829 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.765832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.765876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.765921 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:48.766341 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.767118 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.768929 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.769540 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.769616 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.769680 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.770017 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.770093 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:13:48.770262 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.764861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:43103 (local address 127.18.90.1:51465)
0806 16:13:48.765207 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.765227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.765374 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.766224 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.766228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.766247 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:48.766385 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.766392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.767402 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.767406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.768790 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.768802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.769825 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.769828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.769929 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.769950 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.770021 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.770092 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":5,"thread_start_us":136,"threads_started":1}
I20260806 16:13:48.770699 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.765731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:51465 (local address 127.18.90.2:43103)
0806 16:13:48.765852 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.765857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.765874 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:13:48.765932 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.765934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.765989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.766073 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:13:48.766536 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.767253 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.768944 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.769552 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.769636 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.769711 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.770486 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.770571 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:13:48.770864 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.765312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:46657 (local address 127.18.90.1:43907)
0806 16:13:48.765588 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.765602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:13:48.765720 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:13:48.766049 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:13:48.766051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:13:48.766064 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:13:48.766215 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.766221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.767278 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.767281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.768772 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:13:48.768784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.769777 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.769780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:13:48.769884 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.769907 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:13:48.770641 (+ 734us) client_negotiation.cc:240] Negotiation successful
0806 16:13:48.770712 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":95,"thread_start_us":116,"threads_started":1}
I20260806 16:13:48.771591 30475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ad95ede064812b576d451694e69b1" candidate_uuid: "c5ba6aa4559c45fdb20a077d1349c752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed59d318db34469bee82c6a42c25da8" is_pre_election: true
I20260806 16:13:48.771605 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ad95ede064812b576d451694e69b1" candidate_uuid: "c5ba6aa4559c45fdb20a077d1349c752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd06800dc95482fa29d996726f5da64" is_pre_election: true
I20260806 16:13:48.771908 30475 raft_consensus.cc:2454] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ba6aa4559c45fdb20a077d1349c752 in term 0.
I20260806 16:13:48.771912 30610 raft_consensus.cc:2454] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ba6aa4559c45fdb20a077d1349c752 in term 0.
I20260806 16:13:48.772464 30271 leader_election.cc:302] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ba6aa4559c45fdb20a077d1349c752, dfd06800dc95482fa29d996726f5da64; no voters:
I20260806 16:13:48.772753 30684 raft_consensus.cc:2781] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:13:48.772852 30684 raft_consensus.cc:482] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:13:48.772892 30684 raft_consensus.cc:3037] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:48.774209 30684 raft_consensus.cc:504] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.774564 30684 leader_election.cc:288] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed59d318db34469bee82c6a42c25da8 (127.18.90.2:43103), dfd06800dc95482fa29d996726f5da64 (127.18.90.3:46657)
I20260806 16:13:48.774974 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ad95ede064812b576d451694e69b1" candidate_uuid: "c5ba6aa4559c45fdb20a077d1349c752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd06800dc95482fa29d996726f5da64"
I20260806 16:13:48.774988 30475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ad95ede064812b576d451694e69b1" candidate_uuid: "c5ba6aa4559c45fdb20a077d1349c752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed59d318db34469bee82c6a42c25da8"
I20260806 16:13:48.775113 30610 raft_consensus.cc:3037] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:48.775113 30475 raft_consensus.cc:3037] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:13:48.776432 30475 raft_consensus.cc:2454] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ba6aa4559c45fdb20a077d1349c752 in term 1.
I20260806 16:13:48.776432 30610 raft_consensus.cc:2454] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ba6aa4559c45fdb20a077d1349c752 in term 1.
I20260806 16:13:48.776737 30271 leader_election.cc:302] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed59d318db34469bee82c6a42c25da8, c5ba6aa4559c45fdb20a077d1349c752; no voters:
I20260806 16:13:48.776943 30684 raft_consensus.cc:2781] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:13:48.777215 30684 raft_consensus.cc:686] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [term 1 LEADER]: Becoming Leader. State: Replica: c5ba6aa4559c45fdb20a077d1349c752, State: Running, Role: LEADER
I20260806 16:13:48.777557 30684 consensus_queue.cc:237] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } }
I20260806 16:13:48.781188 30168 catalog_manager.cc:5495] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ba6aa4559c45fdb20a077d1349c752 (127.18.90.1). New cstate: current_term: 1 leader_uuid: "c5ba6aa4559c45fdb20a077d1349c752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ba6aa4559c45fdb20a077d1349c752" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 } health_report { overall_health: UNKNOWN } } }
W20260806 16:13:48.810112 30658 tablet.cc:2369] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:13:48.814759 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.809716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.18.90.1:43941)
0806 16:13:48.809837 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:13:48.809841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:13:48.809894 (+ 53us) server_negotiation.cc:407] Connection header received
0806 16:13:48.810104 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:13:48.810108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:13:48.810182 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:13:48.810284 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:13:48.810683 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.811572 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.812919 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:13:48.813274 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:13:48.813390 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:13:48.813771 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:13:48.814338 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:13:48.814461 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:13:48.814555 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 16:13:48.814618 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:13:48.822119 30610 raft_consensus.cc:1261] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 1 FOLLOWER]: Refusing update from remote peer c5ba6aa4559c45fdb20a077d1349c752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:13:48.822160 30475 raft_consensus.cc:1261] T b88ad95ede064812b576d451694e69b1 P 0ed59d318db34469bee82c6a42c25da8 [term 1 FOLLOWER]: Refusing update from remote peer c5ba6aa4559c45fdb20a077d1349c752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:13:48.822906 30684 consensus_queue.cc:1035] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed59d318db34469bee82c6a42c25da8" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:13:48.823139 30687 consensus_queue.cc:1035] T b88ad95ede064812b576d451694e69b1 P c5ba6aa4559c45fdb20a077d1349c752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd06800dc95482fa29d996726f5da64" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:13:48.835059 30690 mvcc.cc:205] Tried to move back new op lower bound from 7315590466847154176 to 7315590466675879936. Current Snapshot: MvccSnapshot[applied={T|T < 7315590466847154176}]
I20260806 16:13:48.835346 30691 mvcc.cc:205] Tried to move back new op lower bound from 7315590466847154176 to 7315590466675879936. Current Snapshot: MvccSnapshot[applied={T|T < 7315590466847154176}]
I20260806 16:13:48.840139 30692 mvcc.cc:205] Tried to move back new op lower bound from 7315590466847154176 to 7315590466675879936. Current Snapshot: MvccSnapshot[applied={T|T < 7315590466847154176}]
I20260806 16:13:48.850199 30168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:13:48.850411 30168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:13:48.853339 30168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efc84f0c661a4efd9ed17de471e6a0a1: Sending DeleteTablet for 3 replicas of tablet b88ad95ede064812b576d451694e69b1
I20260806 16:13:48.854314 30316 tablet_service.cc:1514] Processing DeleteTablet for tablet b88ad95ede064812b576d451694e69b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:13:48 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:41520
I20260806 16:13:48.854360 30455 tablet_service.cc:1514] Processing DeleteTablet for tablet b88ad95ede064812b576d451694e69b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:13:48 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:45116
I20260806 16:13:48.854452 30590 tablet_service.cc:1514] Processing DeleteTablet for tablet b88ad95ede064812b576d451694e69b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:13:48 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:37940
I20260806 16:13:48.855911 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 30249
I20260806 16:13:48.861235 30702 tablet_replica.cc:330] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: stopping tablet replica
I20260806 16:13:48.864666 30702 raft_consensus.cc:2229] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:13:48.865217 30702 raft_consensus.cc:2258] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:13:48.867163 30702 ts_tablet_manager.cc:1905] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:13:48.868044 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 30390
W20260806 16:13:48.868189 30153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:43941 (error 108)
I20260806 16:13:48.871800 30702 ts_tablet_manager.cc:1918] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:13:48.871917 30702 log.cc:1196] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/b88ad95ede064812b576d451694e69b1
I20260806 16:13:48.872287 30702 ts_tablet_manager.cc:1939] T b88ad95ede064812b576d451694e69b1 P dfd06800dc95482fa29d996726f5da64: Deleting consensus metadata
I20260806 16:13:48.874476 30155 catalog_manager.cc:4841] TS dfd06800dc95482fa29d996726f5da64 (127.18.90.3:46657): tablet b88ad95ede064812b576d451694e69b1 (table test-table [id=f7884d51b42c4226b954ca4f03ed1fcb]) successfully deleted
I20260806 16:13:48.874644 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.872914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43941 (local address 127.0.0.1:41536)
0806 16:13:48.874402 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.874480 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:43941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1398}
W20260806 16:13:48.874820 30153 catalog_manager.cc:4584] TS c5ba6aa4559c45fdb20a077d1349c752 (127.18.90.1:43941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b88ad95ede064812b576d451694e69b1: Network error: Client connection negotiation failed: client connection to 127.18.90.1:43941: connect: Connection refused (error 111)
I20260806 16:13:48.875988 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 30526
I20260806 16:13:48.878319 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:13:48.876210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:43103 (local address 127.0.0.1:45128)
0806 16:13:48.878199 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0806 16:13:48.878247 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.2:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1899}
I20260806 16:13:48.883491 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 30136
I20260806 16:13:48.891732 18792 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 16:13:49.237064 18792 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 16:13:49.237332 18792 mini_ranger.cc:66] Stopping Ranger...
I20260806 16:13:49.584589 18792 mini_ranger.cc:68] Stopped Ranger
2026-08-06 16:13:49.584 UTC [29720] LOG: received smart shutdown request
2026-08-06 16:13:49.588 UTC [29720] LOG: background worker "logical replication launcher" (PID 29729) exited with exit code 1
2026-08-06 16:13:49.590 UTC [29724] LOG: shutting down
2026-08-06 16:13:49.751 UTC [29720] LOG: database system is shut down
2026-08-06T16:13:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46951 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:13:49 dist-test-slave-0b58 krb5kdc[30713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:13:49 dist-test-slave-0b58 krb5kdc[30713](info): set up 2 sockets
Aug 06 16:13:49 dist-test-slave-0b58 krb5kdc[30713](info): commencing operation
krb5kdc: starting...
W20260806 16:13:50.920526 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:13:51 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:13:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/rangeradmin_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/rangeradmin_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/rangerlookup_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/rangerlookup_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
I20260806 16:13:51.095139 18792 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskG62Umc/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/postgres -l logfile start
2026-08-06 16:13:53.505 UTC [30743] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 16:13:53.505 UTC [30743] LOG: listening on IPv4 address "127.18.90.20", port 50503
2026-08-06 16:13:53.509 UTC [30743] LOG: listening on Unix socket "/tmp/.s.PGSQL.50503"
2026-08-06 16:13:53.530 UTC [30746] LOG: database system was shut down at 2026-08-06 16:13:52 UTC
2026-08-06 16:13:53.534 UTC [30743] LOG: database system is ready to accept connections
I20260806 16:13:54.476346 18792 mini_postgres.cc:96] Postgres bound to 50503
2026-08-06 16:13:54.483 UTC [30754] FATAL: database "slave" does not exist
127.18.90.20:50503 - accepting connections
I20260806 16:13:54.483618 18792 mini_ranger.cc:161] Starting Ranger...
I20260806 16:13:54.499013 18792 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 16:13:54.941437 18792 mini_ranger.cc:87] Created ranger Postgres database
I20260806 16:13:54.941530 18792 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-admin
2026-08-06 16:13:56,019 [I] DB FLAVOR :POSTGRES
2026-08-06 16:13:56,020 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:13:56,020 [I] Checking connection..
2026-08-06 16:13:56,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 16:13:56,281 [I] Checking connection passed.
2026-08-06 16:13:56,282 [I] --------- Verifying version history table ---------
2026-08-06 16:13:56,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:13:56,546 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:13:56,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:13:56,784 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:13:56,784 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 16:13:56,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 16:13:57.024 UTC [30818] WARNING: there is no transaction in progress
2026-08-06 16:13:57,031 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 16:13:57,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:13:57,262 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 16:13:57,263 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:13:57,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 16:13:57,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 16:13:57,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0b58.c.gcp-upstream.internal','N') ;"
2026-08-06 16:13:57,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:13:58,230 [I] Table x_portal_user does not exist in database ranger
2026-08-06 16:13:58,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:13:58,485 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 16:13:58,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:13:58,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:13:58,993 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 16:13:58,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 16:13:59.938 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:13:59.954 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:13:59.969 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:13:59.984 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.000 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.021 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.075 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.081 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.095 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.105 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.115 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.130 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.139 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.146 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.153 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00.159 UTC [30928] WARNING: there is no transaction in progress
2026-08-06 16:14:00,792 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 16:14:00,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 16:14:01,052 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 16:14:01,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:14:01,310 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 16:14:01,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:14:01,580 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 16:14:01,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:14:01,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:14:02,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0b58.c.gcp-upstream.internal';"
2026-08-06 16:14:02,348 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 16:14:02,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 16:14:02,616 [I] DB_PATCHES have already been applied
I20260806 16:14:02.622159 18792 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-admin:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskG62Umc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 16:14:02.622226 18792 mini_ranger.cc:193] Using host: 127.18.90.20
I20260806 16:14:02.626592 18792 mini_ranger.cc:227] Ranger admin URL: http://127.18.90.20:40489
Aug 06, 2026 4:14:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:14:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 16:14:03.725643 18792 mini_ranger.cc:160] Time spent starting Ranger: real 9.242s user 0.002s sys 0.010s
Aug 06, 2026 4:14:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:14:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 4:14:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:14:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:14:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 16:14:25.988853 18792 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.90.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.90.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/rangerkms_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.90.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/rangerkms_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.90.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.90.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.90.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/HTTP_127.18.90.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 16:14:26 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 16:14:26.091885 18792 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 16:14:26.103411 18792 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 16:14:26.922145 18792 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 16:14:26.922238 18792 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms
I20260806 16:14:26.922261 18792 mini_ranger_kms.cc:227] Using postgres at 127.18.90.20:50503
2026-08-06 16:14:27,110 [I] DB FLAVOR :POSTGRES
2026-08-06 16:14:27,110 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:14:27,110 [I] Checking connection
2026-08-06 16:14:27,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 16:14:27,379 [I] connection success
2026-08-06 16:14:27,379 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 16:14:27,379 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 16:14:27,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 16:14:27,643 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 16:14:27,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:14:27,644 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 16:14:27,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.90.20:50503/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 16:14:27,934 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 16:14:28.389616 18792 mini_ranger_kms.cc:310] Created kms service
I20260806 16:14:28.607132 18792 mini_ranger_kms.cc:326] Created kudu user
I20260806 16:14:28.654976 18792 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 16:14:28 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:29.051903 18792 mini_ranger_kms.cc:384] Added ranger policy
I20260806 16:14:29.052062 18792 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms:/tmp/dist-test-taskG62Umc/build/debug/bin/postgresql.jar:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskG62Umc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 16:14:29.052083 18792 mini_ranger_kms.cc:240] Using host: 127.18.90.20
I20260806 16:14:29.057159 18792 mini_ranger_kms.cc:276] Ranger KMS PID: 31109
I20260806 16:14:29.057257 18792 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.90.20:58861
Aug 06, 2026 4:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 16:14:31 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.90.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:31.332051 18792 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.240s user 0.011s sys 0.161s
I20260806 16:14:31.332197 18792 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 16:14:31.332319 18792 mini_ranger_kms.cc:401] 127.18.90.20:58861/kms/v1/keys
Aug 06 16:14:31 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.90.20@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
Aug 06 16:14:32 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.90.20@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
Aug 06 16:14:34 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:14:34.356457 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44153
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:44153
--ranger_config_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:34.471261 31161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:34.471536 31161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:34.471575 31161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:34.475860 31161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:34.475931 31161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:34.475951 31161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:34.475968 31161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:34.475984 31161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:34.481760 31161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.90.20:58861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:44153
--ranger_config_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:34.482908 31161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:34.483979 31161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:14:34.493104 31161 server_base.cc:1034] running on GCE node
W20260806 16:14:34.493039 31166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:34.493039 31167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:34.493067 31169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:34.494038 31161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:34.495262 31161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:34.496445 31161 hybrid_clock.cc:648] HybridClock initialized: now 1786032874496427 us; error 38 us; skew 500 ppm
Aug 06 16:14:34 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:34.499259 31161 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:34.500320 31161 webserver.cc:460] Webserver started at http://127.18.90.62:34421/ using document root <none> and password file <none>
I20260806 16:14:34.500888 31161 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:34.500945 31161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:34.501180 31161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:14:34 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:34.555943 31161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "b539c9755419467e95c854f836c35799"
format_stamp: "Formatted at 2026-08-06 16:14:34 on dist-test-slave-0b58"
server_key: "86e7e38b131fd8a400fa5b1ead02d7cf"
server_key_iv: "e65f5f30d4825d3428d567a504b83207"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:34.556648 31161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "b539c9755419467e95c854f836c35799"
format_stamp: "Formatted at 2026-08-06 16:14:34 on dist-test-slave-0b58"
server_key: "86e7e38b131fd8a400fa5b1ead02d7cf"
server_key_iv: "e65f5f30d4825d3428d567a504b83207"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:34.560834 31161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:14:34.578174 31176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:34.579885 31161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:14:34.580034 31161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "b539c9755419467e95c854f836c35799"
format_stamp: "Formatted at 2026-08-06 16:14:34 on dist-test-slave-0b58"
server_key: "86e7e38b131fd8a400fa5b1ead02d7cf"
server_key_iv: "e65f5f30d4825d3428d567a504b83207"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:34.580238 31161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:34.596608 31161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:34.599794 31161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:34.600003 31161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:34.608616 31161 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:44153
I20260806 16:14:34.608637 31228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:44153 every 8 connection(s)
I20260806 16:14:34.609809 31161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:34.612759 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:34.614295 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31161
I20260806 16:14:34.614423 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:14:34.614760 18792 external_mini_cluster.cc:1428] Setting key accdc9a13935f28e2ad071348728fde5
I20260806 16:14:34.620062 31229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Bootstrap starting.
I20260806 16:14:34.622714 31229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:34.623667 31229 log.cc:824] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:14:34 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:34.626122 31229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: No bootstrap required, opened a new log
I20260806 16:14:34.630568 31229 raft_consensus.cc:348] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:34.630908 31229 raft_consensus.cc:374] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:34.631001 31229 raft_consensus.cc:729] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b539c9755419467e95c854f836c35799, State: Initialized, Role: FOLLOWER
I20260806 16:14:34.631481 31229 consensus_queue.cc:260] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:34.631635 31229 raft_consensus.cc:388] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:14:34.631728 31229 raft_consensus.cc:482] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:14:34.631855 31229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:34.632018 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:34.616348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.18.90.62:44153)
0806 16:14:34.616828 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:34.616848 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:34.616894 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:14:34.617758 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:34.617782 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:34.618213 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:34.618577 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:34.619547 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:34.621101 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:34.622307 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:34.623011 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:34.623179 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:34.626382 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:34.626389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:34.626408 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:34.626439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:34.628497 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:34.629010 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:34.629017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:34.629023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:34.629156 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:34.629560 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:34.629564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:34.629566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:34.630086 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:34.630276 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:34.630611 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 16:14:34.631071 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260806 16:14:34.632954 31229 raft_consensus.cc:504] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:34.633307 31229 leader_election.cc:302] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b539c9755419467e95c854f836c35799; no voters:
I20260806 16:14:34.633705 31229 leader_election.cc:288] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:14:34.633800 31234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:34.634038 31234 raft_consensus.cc:686] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 LEADER]: Becoming Leader. State: Replica: b539c9755419467e95c854f836c35799, State: Running, Role: LEADER
I20260806 16:14:34.634387 31234 consensus_queue.cc:237] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:34.635066 31229 sys_catalog.cc:564] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:34.635599 31234 sys_catalog.cc:455] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b539c9755419467e95c854f836c35799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } } }
I20260806 16:14:34.635792 31234 sys_catalog.cc:458] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:34.636068 31234 sys_catalog.cc:455] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b539c9755419467e95c854f836c35799. Latest consensus state: current_term: 1 leader_uuid: "b539c9755419467e95c854f836c35799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } } }
I20260806 16:14:34.636232 31234 sys_catalog.cc:458] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:34.637461 31241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:34.637573 31229 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 16:14:34.643440 31241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:34.652235 31241 catalog_manager.cc:1347] Generated new cluster ID: 312dae08fd33476dac35e3afa3fc147a
I20260806 16:14:34.652384 31241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:14:34.671684 31241 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:14:34.673164 31241 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:14:34.688823 31241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Generated new TSK 0
I20260806 16:14:34.689818 31241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:14:35.407927 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44153
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:35.541503 31275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:35.541821 31275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:35.541949 31275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:35.546257 31275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:35.546348 31275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:35.546437 31275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:14:35.552011 31275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.90.20:58861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:35.553217 31275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:35.554463 31275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:35.565964 31280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:35.565999 31283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:35.566620 31281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:35.566954 31275 server_base.cc:1034] running on GCE node
I20260806 16:14:35.567430 31275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:35.568222 31275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:35.569489 31275 hybrid_clock.cc:648] HybridClock initialized: now 1786032875569465 us; error 39 us; skew 500 ppm
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:35.573179 31275 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:14:35.574393 31275 webserver.cc:460] Webserver started at http://127.18.90.1:33249/ using document root <none> and password file <none>
I20260806 16:14:35.575170 31275 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:35.575332 31275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:35.575596 31275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:35.592296 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "b883cba0a2db4bc9886972ab23d8432e"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "440a1908b9d3a9ad56cd32c4cf93f764"
server_key_iv: "d3d0aabe5720be2325a0d284052bfe7e"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:35.593153 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "b883cba0a2db4bc9886972ab23d8432e"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "440a1908b9d3a9ad56cd32c4cf93f764"
server_key_iv: "d3d0aabe5720be2325a0d284052bfe7e"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:35.602797 31275 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.005s
I20260806 16:14:35.624385 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:35.626153 31275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:14:35.626287 31275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "b883cba0a2db4bc9886972ab23d8432e"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "440a1908b9d3a9ad56cd32c4cf93f764"
server_key_iv: "d3d0aabe5720be2325a0d284052bfe7e"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:35.626425 31275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:35.658708 31275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:35.662050 31275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:35.662240 31275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:35.662935 31275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:35.663859 31275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:35.663908 31275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:35.663949 31275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:35.664002 31275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:35.674958 31275 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:45117
I20260806 16:14:35.675055 31406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:45117 every 8 connection(s)
I20260806 16:14:35.676329 31275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:14:35.680963 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31275
I20260806 16:14:35.681092 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:14:35.681424 18792 external_mini_cluster.cc:1428] Setting key 6e20332293f983877ce718eee5b9dd4e
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:35.710753 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:35.684174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:54295 (local address 127.18.90.62:44153)
0806 16:14:35.684500 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:35.684511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:35.695893 (+ 11382us) server_negotiation.cc:407] Connection header received
0806 16:14:35.697339 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:35.697347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:35.697425 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:35.697499 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:35.699442 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.700346 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.702141 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.702525 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.702602 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:35.705890 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:35.705894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:35.705897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:35.705928 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:35.707845 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:35.708520 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:35.708527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:35.708531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:35.708633 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:35.709086 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:35.709092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:35.709096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:35.709384 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:35.709484 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:35.710293 (+ 809us) server_negotiation.cc:299] Negotiation successful
0806 16:14:35.710464 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
I20260806 16:14:35.712772 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:35.683061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44153 (local address 127.18.90.1:54295)
0806 16:14:35.695684 (+ 12623us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:35.695747 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:35.697005 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:35.698021 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:35.698034 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:35.698451 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:35.699230 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:35.699256 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.700510 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.700514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:35.701992 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:35.702004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.702849 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.702853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:35.703068 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:35.703834 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:35.703856 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:35.705664 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:35.707974 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:35.707985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:35.708000 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:35.708350 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:35.708753 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:35.708759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:35.708763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:35.708969 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:35.709565 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:35.709574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:35.709940 (+ 366us) client_negotiation.cc:769] Sending connection context
0806 16:14:35.711086 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0806 16:14:35.711514 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12342,"thread_start_us":146,"threads_started":1}
I20260806 16:14:35.714917 31407 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44153
I20260806 16:14:35.715193 31407 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:35.715871 31407 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:35.717798 31193 ts_manager.cc:194] Registered new tserver with Master: b883cba0a2db4bc9886972ab23d8432e (127.18.90.1:45117)
I20260806 16:14:35.719442 31193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:54295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:14:35.727450 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44153
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 16:14:35.860474 31415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:35.860770 31415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:35.860877 31415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:35.866199 31415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:35.866300 31415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:35.866398 31415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:14:35.872437 31415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.90.20:58861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:35.873637 31415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:35.874735 31415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:35.884634 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:35.885097 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:35.885425 31415 server_base.cc:1034] running on GCE node
W20260806 16:14:35.885637 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:35.886121 31415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:35.886889 31415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:35.888088 31415 hybrid_clock.cc:648] HybridClock initialized: now 1786032875888067 us; error 37 us; skew 500 ppm
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:35.891695 31415 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:14:35.892817 31415 webserver.cc:460] Webserver started at http://127.18.90.2:45755/ using document root <none> and password file <none>
I20260806 16:14:35.893560 31415 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:35.893661 31415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:35.893970 31415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:35.910295 31415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "ef96ccc8d2554fbaad0830e1a344b7c2"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "7794f5382f2ce180106c704eec83150b"
server_key_iv: "a0c9517cf2ee4727b5f1a8f8fee40969"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:35.911049 31415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "ef96ccc8d2554fbaad0830e1a344b7c2"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "7794f5382f2ce180106c704eec83150b"
server_key_iv: "a0c9517cf2ee4727b5f1a8f8fee40969"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:35.916200 31415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 16:14:35.936746 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:35.938498 31415 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 16:14:35.938737 31415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "ef96ccc8d2554fbaad0830e1a344b7c2"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "7794f5382f2ce180106c704eec83150b"
server_key_iv: "a0c9517cf2ee4727b5f1a8f8fee40969"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:35.939005 31415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:35.953382 31415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:35.957123 31415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:35.957393 31415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:35.958245 31415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:35.959460 31415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:35.959574 31415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:35.959697 31415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:35.959769 31415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:35.971131 31415 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:37157
I20260806 16:14:35.972409 31415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:14:35.972790 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31415
I20260806 16:14:35.972906 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:14:35.973199 18792 external_mini_cluster.cc:1428] Setting key 5dbedf120506cbaa3a465a64c6a93f21
I20260806 16:14:35.973338 31545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:37157 every 8 connection(s)
Aug 06 16:14:35 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:14:35.995064 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:35.977434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:33115 (local address 127.18.90.62:44153)
0806 16:14:35.977567 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:35.977574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:35.977953 (+ 379us) server_negotiation.cc:407] Connection header received
0806 16:14:35.979285 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:35.979289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:35.979347 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:35.979405 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:35.981321 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.982235 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.984647 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.985013 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.985098 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:35.990669 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:35.990676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:35.990681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:35.990715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:35.992605 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:35.993216 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:35.993220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:35.993222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:35.993316 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:35.993779 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:35.993782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:35.993784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:35.993998 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:35.994106 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:35.994731 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 16:14:35.994861 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:14:35.998220 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:35.976382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44153 (local address 127.18.90.2:33115)
0806 16:14:35.977789 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:35.977854 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:35.979018 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:35.979863 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:35.979872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:35.980251 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:35.981021 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:35.981044 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.982415 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.982419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:35.983842 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:35.983854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:35.985323 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:35.985326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:35.985528 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:35.988442 (+ 2914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:35.988470 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:35.990439 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:35.992826 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:35.992830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:35.992844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:35.993093 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:35.993442 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:35.993445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:35.993446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:35.993663 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:35.994105 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:35.994112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:35.994389 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 16:14:35.994678 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 16:14:35.995070 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":141,"threads_started":1}
I20260806 16:14:35.999917 31546 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44153
I20260806 16:14:36.000169 31546 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:36.000836 31546 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:36.002207 31193 ts_manager.cc:194] Registered new tserver with Master: ef96ccc8d2554fbaad0830e1a344b7c2 (127.18.90.2:37157)
I20260806 16:14:36.003111 31193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:33115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:14:36.015144 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44153
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:36.175109 31551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:36.175549 31551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:36.175622 31551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:36.181938 31551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:36.182086 31551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:36.182191 31551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:14:36.190573 31551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.90.20:58861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:36.192283 31551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:36.193751 31551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:36.207022 31559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:36.207202 31556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:36.207521 31551 server_base.cc:1034] running on GCE node
W20260806 16:14:36.207527 31557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:36.208927 31551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:36.209762 31551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:36.211062 31551 hybrid_clock.cc:648] HybridClock initialized: now 1786032876210989 us; error 92 us; skew 500 ppm
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:36.215157 31551 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:14:36.216279 31551 webserver.cc:460] Webserver started at http://127.18.90.3:38275/ using document root <none> and password file <none>
I20260806 16:14:36.217118 31551 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:36.217206 31551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:36.217489 31551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:36.245443 31551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "70eb9babc0334af18283068795deecf5"
format_stamp: "Formatted at 2026-08-06 16:14:36 on dist-test-slave-0b58"
server_key: "0661217f33662ad445ca8503a9406dbd"
server_key_iv: "efcaa7978571e3ae5630803349fd7cbc"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:36.246403 31551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "70eb9babc0334af18283068795deecf5"
format_stamp: "Formatted at 2026-08-06 16:14:36 on dist-test-slave-0b58"
server_key: "0661217f33662ad445ca8503a9406dbd"
server_key_iv: "efcaa7978571e3ae5630803349fd7cbc"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:36.254022 31551 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.002s
I20260806 16:14:36.281116 31566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.283006 31551 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260806 16:14:36.283145 31551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "70eb9babc0334af18283068795deecf5"
format_stamp: "Formatted at 2026-08-06 16:14:36 on dist-test-slave-0b58"
server_key: "0661217f33662ad445ca8503a9406dbd"
server_key_iv: "efcaa7978571e3ae5630803349fd7cbc"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:36.283288 31551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032875, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:36.322044 31551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:36.326237 31551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:36.326538 31551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:36.327461 31551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:36.328687 31551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:36.328800 31551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.328991 31551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:36.329066 31551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.354331 31551 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:39345
I20260806 16:14:36.355624 31551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:14:36.356442 31680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:39345 every 8 connection(s)
I20260806 16:14:36.359570 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31551
I20260806 16:14:36.359702 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:14:36.360044 18792 external_mini_cluster.cc:1428] Setting key 2c4b0b55194c00fe6fe0af29836a4797
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:36.374683 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:36.360718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:43461 (local address 127.18.90.62:44153)
0806 16:14:36.360844 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:36.360848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:36.362930 (+ 2082us) server_negotiation.cc:407] Connection header received
0806 16:14:36.363872 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:36.363875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:36.363927 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:36.364001 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:36.365273 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.365812 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.366992 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.367274 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.367386 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:36.369880 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:36.369884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:36.369886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:36.369920 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:36.371434 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:36.371990 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:36.371994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:36.371995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:36.372068 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:36.372495 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:36.372498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:36.372500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:36.372736 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:36.372821 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:36.374378 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0806 16:14:36.374512 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 16:14:36.375689 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:36.359517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44153 (local address 127.18.90.3:43461)
0806 16:14:36.362753 (+ 3236us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:36.362811 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:36.363672 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:36.364212 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:36.364219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:36.364620 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:36.365120 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:36.365133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.365942 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.365946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:36.366863 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:36.366872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.367433 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.367437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:36.367618 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:36.368304 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:36.368326 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:36.369709 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:36.371572 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:36.371578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:36.371594 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:36.371870 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:36.372159 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:36.372163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:36.372165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:36.372388 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:36.372957 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:36.372964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:36.373760 (+ 796us) client_negotiation.cc:769] Sending connection context
0806 16:14:36.374321 (+ 561us) client_negotiation.cc:240] Negotiation successful
0806 16:14:36.374653 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3000,"thread_start_us":154,"threads_started":1}
I20260806 16:14:36.376820 31681 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44153
I20260806 16:14:36.377070 31681 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:36.377573 31681 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:36.378613 31193 ts_manager.cc:194] Registered new tserver with Master: 70eb9babc0334af18283068795deecf5 (127.18.90.3:39345)
I20260806 16:14:36.379201 31193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:43461
I20260806 16:14:36.385983 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:14:36.386672 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31275
I20260806 16:14:36.393944 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31415
I20260806 16:14:36.402282 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31551
I20260806 16:14:36.408084 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31161
I20260806 16:14:36.414551 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44153
--webserver_interface=127.18.90.62
--webserver_port=34421
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:44153
--ranger_config_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:36.527747 31684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:36.528010 31684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:36.528067 31684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:36.532240 31684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:36.532346 31684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:36.532366 31684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:36.532382 31684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:36.532431 31684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:36.538040 31684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:44153
--ranger_config_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=34421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:36.539151 31684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:36.540205 31684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:36.548667 31689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:36.548856 31684 server_base.cc:1034] running on GCE node
W20260806 16:14:36.548689 31692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:36.548667 31690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:36.549629 31684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:36.550732 31684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:36.551926 31684 hybrid_clock.cc:648] HybridClock initialized: now 1786032876551911 us; error 25 us; skew 500 ppm
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:36.554793 31684 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:36.555683 31684 webserver.cc:460] Webserver started at http://127.18.90.62:34421/ using document root <none> and password file <none>
I20260806 16:14:36.556232 31684 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:36.556391 31684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:36.604974 31684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:14:36.607283 31700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.608636 31684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 16:14:36.608807 31684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "b539c9755419467e95c854f836c35799"
format_stamp: "Formatted at 2026-08-06 16:14:34 on dist-test-slave-0b58"
server_key: "86e7e38b131fd8a400fa5b1ead02d7cf"
server_key_iv: "e65f5f30d4825d3428d567a504b83207"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:36.609452 31684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:36.623811 31684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:36.626559 31684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:36.626780 31684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:36.634102 31684 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:44153
I20260806 16:14:36.634109 31752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:44153 every 8 connection(s)
I20260806 16:14:36.635203 31684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:36.639513 31753 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:14:36.641185 31753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Bootstrap starting.
I20260806 16:14:36.642201 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31684
I20260806 16:14:36.642647 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:45117
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:44153
--webserver_port=33249
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:36.657836 31753 log.cc:824] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:36.664619 31753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:14:36.665086 31753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Bootstrap complete.
I20260806 16:14:36.669366 31753 raft_consensus.cc:348] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:36.669778 31753 raft_consensus.cc:729] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b539c9755419467e95c854f836c35799, State: Initialized, Role: FOLLOWER
I20260806 16:14:36.670316 31753 consensus_queue.cc:260] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:36.670466 31753 raft_consensus.cc:388] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:14:36.670557 31753 raft_consensus.cc:482] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:14:36.670676 31753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:14:36.672686 31753 raft_consensus.cc:504] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:36.673032 31753 leader_election.cc:302] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b539c9755419467e95c854f836c35799; no voters:
I20260806 16:14:36.673394 31753 leader_election.cc:288] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:14:36.673508 31757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:14:36.673780 31757 raft_consensus.cc:686] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [term 2 LEADER]: Becoming Leader. State: Replica: b539c9755419467e95c854f836c35799, State: Running, Role: LEADER
I20260806 16:14:36.674134 31757 consensus_queue.cc:237] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } }
I20260806 16:14:36.674646 31753 sys_catalog.cc:564] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:36.675488 31758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b539c9755419467e95c854f836c35799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } } }
I20260806 16:14:36.675612 31758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:36.676016 31759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b539c9755419467e95c854f836c35799. Latest consensus state: current_term: 2 leader_uuid: "b539c9755419467e95c854f836c35799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b539c9755419467e95c854f836c35799" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44153 } } }
I20260806 16:14:36.676159 31759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:36.676545 31766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:36.677101 31753 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 16:14:36.683017 31766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:36.683910 31766 catalog_manager.cc:1259] Loaded cluster ID: 312dae08fd33476dac35e3afa3fc147a
I20260806 16:14:36.684015 31766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:14:36.686733 31766 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:14:36.687528 31766 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Loaded TSK: 0
I20260806 16:14:36.688915 31766 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:14:36.765323 31755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:36.765609 31755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:36.765710 31755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:36.770623 31755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:36.770715 31755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:36.770817 31755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:14:36.776726 31755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:45117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=33249
--tserver_master_addrs=127.18.90.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:36.777918 31755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:36.779070 31755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:36.791539 31790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:36.791536 31793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:36.792171 31791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:36.792452 31755 server_base.cc:1034] running on GCE node
I20260806 16:14:36.792894 31755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:36.793522 31755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:36.794682 31755 hybrid_clock.cc:648] HybridClock initialized: now 1786032876794650 us; error 47 us; skew 500 ppm
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:36.797713 31755 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:14:36.798606 31755 webserver.cc:460] Webserver started at http://127.18.90.1:33249/ using document root <none> and password file <none>
I20260806 16:14:36.799356 31755 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:36.799465 31755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:36.829752 31755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:14:36.839592 31801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.841454 31755 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260806 16:14:36.841593 31755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "b883cba0a2db4bc9886972ab23d8432e"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "440a1908b9d3a9ad56cd32c4cf93f764"
server_key_iv: "d3d0aabe5720be2325a0d284052bfe7e"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:36.842370 31755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:36.858500 31755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:36.861681 31755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:36.861903 31755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:36.862684 31755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:36.863665 31755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:36.863740 31755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.863840 31755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:36.863878 31755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:36.875270 31755 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:45117
I20260806 16:14:36.877007 31755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:14:36.880532 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31755
I20260806 16:14:36.881078 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:37157
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:44153
--webserver_port=45755
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:36.884984 31914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:45117 every 8 connection(s)
Aug 06 16:14:36 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:36.913618 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:36.879501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:51417 (local address 127.18.90.62:44153)
0806 16:14:36.884455 (+ 4954us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:36.884481 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:36.884535 (+ 54us) server_negotiation.cc:407] Connection header received
0806 16:14:36.887210 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:36.887231 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:36.887707 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:36.888092 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:36.889966 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.891520 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.894367 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.895251 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.895450 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:36.906733 (+ 11283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:36.906766 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:36.906792 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:36.906839 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:36.909168 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:36.909946 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:36.909953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:36.909959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:36.910113 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:36.910559 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:36.910562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:36.910564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:36.911112 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:36.911246 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:36.912083 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 16:14:36.912458 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4769,"thread_start_us":122,"threads_started":1}
I20260806 16:14:36.915974 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:36.879657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44153 (local address 127.18.90.1:51417)
0806 16:14:36.880177 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:36.883191 (+ 3014us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:36.884451 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:36.888681 (+ 4230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:36.888691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:36.889067 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:36.889744 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:36.889767 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.892493 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.892501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:36.894179 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:36.894197 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:36.895748 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:36.895755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:36.895999 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:36.896866 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:36.896893 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:36.906384 (+ 9491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:36.909343 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:36.909350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:36.909365 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:36.909787 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:36.910241 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:36.910244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:36.910247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:36.910436 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:36.911321 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:36.911328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:36.911701 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 16:14:36.914531 (+ 2830us) client_negotiation.cc:240] Negotiation successful
0806 16:14:36.914917 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260806 16:14:36.919272 31915 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44153
I20260806 16:14:36.919523 31915 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:36.920115 31915 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
W20260806 16:14:36.922438 31915 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:44153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 16:14:36.922570 31915 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:36.922767 31915 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:36.923638 31915 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:36.923847 31915 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
W20260806 16:14:36.924664 31915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:14:37.058784 31919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:37.059158 31919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:37.059223 31919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:37.065508 31919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:37.065598 31919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:37.065701 31919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:14:37.073853 31919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:37157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=45755
--tserver_master_addrs=127.18.90.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:37.075311 31919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:37.076727 31919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:37.088477 31927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:37.088466 31925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:37.088645 31924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:37.090735 31919 server_base.cc:1034] running on GCE node
I20260806 16:14:37.091174 31919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:37.091807 31919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:37.092988 31919 hybrid_clock.cc:648] HybridClock initialized: now 1786032877092952 us; error 56 us; skew 500 ppm
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:37.095800 31919 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:14:37.096832 31919 webserver.cc:460] Webserver started at http://127.18.90.2:45755/ using document root <none> and password file <none>
I20260806 16:14:37.097417 31919 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:37.097522 31919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:37.119053 31919 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:14:37.122285 31935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:37.123953 31919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:14:37.124094 31919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "ef96ccc8d2554fbaad0830e1a344b7c2"
format_stamp: "Formatted at 2026-08-06 16:14:35 on dist-test-slave-0b58"
server_key: "7794f5382f2ce180106c704eec83150b"
server_key_iv: "a0c9517cf2ee4727b5f1a8f8fee40969"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:37.124933 31919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:37.145248 31919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:37.149170 31919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:37.149438 31919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:37.150249 31919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:37.151371 31919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:37.151504 31919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:37.151590 31919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:37.151664 31919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:37.163010 31919 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:37157
I20260806 16:14:37.163166 32048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:37157 every 8 connection(s)
I20260806 16:14:37.164696 31919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:14:37.172217 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 31919
I20260806 16:14:37.172763 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.90.20:58861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:39345
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:44153
--webserver_port=38275
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:37.186255 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:37.170580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:38803 (local address 127.18.90.62:44153)
0806 16:14:37.170719 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:37.170723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:37.171658 (+ 935us) server_negotiation.cc:407] Connection header received
0806 16:14:37.173137 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:37.173144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:37.173217 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:37.173276 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:37.175212 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.176108 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.177840 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.178241 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.178374 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:37.181484 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:37.181488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:37.181491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:37.181521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:37.183473 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:37.184154 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:37.184158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:37.184160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:37.184293 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:37.184742 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:37.184745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:37.184747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:37.184999 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:37.185104 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:37.185835 (+ 731us) server_negotiation.cc:299] Negotiation successful
0806 16:14:37.185993 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:14:37.188371 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:37.171032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44153 (local address 127.18.90.2:38803)
0806 16:14:37.171508 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:37.171559 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:37.172785 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:37.173744 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:37.173755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:37.174139 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:37.175015 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:37.175041 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.176373 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.176379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:37.177691 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:37.177701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.178569 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.178576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:37.178797 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:37.179555 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:37.179582 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:37.181242 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:37.183606 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:37.183614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:37.183634 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:37.184017 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:37.184401 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:37.184406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:37.184410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:37.184608 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:37.185126 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:37.185135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:37.185510 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 16:14:37.186738 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0806 16:14:37.187114 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":134,"threads_started":1}
I20260806 16:14:37.189952 32049 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44153
I20260806 16:14:37.190223 32049 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.190652 32049 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
W20260806 16:14:37.192133 32049 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:44153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 16:14:37.192291 32049 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.192478 32049 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:37.193290 32049 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.193485 32049 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
W20260806 16:14:37.194217 32049 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:14:37.316560 32052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:37.317003 32052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:37.317097 32052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:37.322979 32052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:37.323068 32052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:37.323161 32052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:14:37.330096 32052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.90.20:58861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:39345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=38275
--tserver_master_addrs=127.18.90.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:37.331346 32052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:37.332592 32052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:37.342602 32058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:37.342741 32061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:37.342602 32059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:37.343979 32052 server_base.cc:1034] running on GCE node
I20260806 16:14:37.344498 32052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:37.345266 32052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:37.346446 32052 hybrid_clock.cc:648] HybridClock initialized: now 1786032877346415 us; error 46 us; skew 500 ppm
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:37.350268 32052 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:14:37.351274 32052 webserver.cc:460] Webserver started at http://127.18.90.3:38275/ using document root <none> and password file <none>
I20260806 16:14:37.352029 32052 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:37.352162 32052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:37.371771 32052 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 16:14:37.375077 32069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:37.376852 32052 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:14:37.376993 32052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "70eb9babc0334af18283068795deecf5"
format_stamp: "Formatted at 2026-08-06 16:14:36 on dist-test-slave-0b58"
server_key: "0661217f33662ad445ca8503a9406dbd"
server_key_iv: "efcaa7978571e3ae5630803349fd7cbc"
server_key_version: "kuduclusterkey@0"
I20260806 16:14:37.377745 32052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:37.407106 32052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:37.411226 32052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:37.411451 32052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:37.412147 32052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:37.413157 32052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:37.413237 32052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:37.413312 32052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:37.413363 32052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:37.424122 32052 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:39345
I20260806 16:14:37.424348 32182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:39345 every 8 connection(s)
I20260806 16:14:37.425513 32052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:14:37.434001 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 32052
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:37.444451 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:37.428795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44153 (local address 127.18.90.3:47405)
0806 16:14:37.429416 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:37.429476 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:37.430784 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:37.431701 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:37.431710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:37.432049 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:37.432770 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:37.432793 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.434006 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.434010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:37.435797 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:37.435809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.436630 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.436633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:37.436844 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:37.437506 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:37.437527 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:37.439249 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:37.441222 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:37.441229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:37.441246 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:37.441592 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:37.441971 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:37.441974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:37.441976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:37.442120 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:37.442517 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:37.442525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:37.442866 (+ 341us) client_negotiation.cc:769] Sending connection context
0806 16:14:37.443095 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 16:14:37.443432 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":186,"threads_started":1}
I20260806 16:14:37.445199 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:37.428226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:47405 (local address 127.18.90.62:44153)
0806 16:14:37.428395 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:37.428400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:37.429591 (+ 1191us) server_negotiation.cc:407] Connection header received
0806 16:14:37.431059 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:37.431063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:37.431125 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:37.431215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:37.432965 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.433840 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.435946 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.436333 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.436415 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:37.439442 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:37.439446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:37.439449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:37.439474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:37.441085 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:37.441731 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:37.441738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:37.441740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:37.441870 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:37.442228 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:37.442231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:37.442233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:37.442411 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:37.442517 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:37.444880 (+ 2363us) server_negotiation.cc:299] Negotiation successful
0806 16:14:37.445009 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 16:14:37.447387 32183 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44153
I20260806 16:14:37.447759 32183 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.448454 32183 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
W20260806 16:14:37.450173 32183 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:44153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 16:14:37.450354 32183 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.450565 32183 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:37.451363 32183 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.451565 32183 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
W20260806 16:14:37.452310 32183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:14:37.454206 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:37.434692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.18.90.62:44153)
0806 16:14:37.437290 (+ 2598us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:37.437296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:37.438459 (+ 1163us) server_negotiation.cc:407] Connection header received
0806 16:14:37.438674 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:37.438677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:37.438727 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:37.438789 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:37.441807 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.442582 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.443727 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:37.444070 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:37.444187 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:37.445967 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:37.445972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:37.445975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:37.446024 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:37.447885 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:37.452887 (+ 5002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:37.452890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:37.452892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:37.452987 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:37.453290 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:37.453293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:37.453295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:37.453483 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:37.453698 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:37.453868 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 16:14:37.453994 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2507,"thread_start_us":99,"threads_started":1}
Aug 06 16:14:37 dist-test-slave-0b58 krb5kdc[30713](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:37.924918 31915 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:37.925275 31915 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:37.926573 31716 ts_manager.cc:194] Registered new tserver with Master: b883cba0a2db4bc9886972ab23d8432e (127.18.90.1:45117)
I20260806 16:14:37.928330 31716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:51417
I20260806 16:14:38.195813 32049 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:38.196137 32049 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:38.197166 31716 ts_manager.cc:194] Registered new tserver with Master: ef96ccc8d2554fbaad0830e1a344b7c2 (127.18.90.2:37157)
I20260806 16:14:38.198006 31716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:38803
Aug 06 16:14:38 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:38.452571 32183 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:38.452857 32183 heartbeater.cc:511] Master 127.18.90.62:44153 requested a full tablet report, sending...
I20260806 16:14:38.453737 31716 ts_manager.cc:194] Registered new tserver with Master: 70eb9babc0334af18283068795deecf5 (127.18.90.3:39345)
I20260806 16:14:38.454536 31716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:47405
Aug 06 16:14:38 dist-test-slave-0b58 krb5kdc[30713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for HTTP/127.18.90.20@KRBTEST.COM
I20260806 16:14:38.469126 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:14:38.493925 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.473375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.18.90.62:44153)
0806 16:14:38.484483 (+ 11108us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.484489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.484506 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:14:38.484563 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.484566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.484620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.484724 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:38.485731 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.486547 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.487751 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.488073 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.488160 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.489928 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:38.489932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:38.489935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:38.489963 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:38.491668 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:38.492129 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:38.492133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:38.492135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:38.492220 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:38.492538 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:38.492541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:38.492543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:38.492766 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:38.492882 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.493600 (+ 718us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.493735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10998,"thread_start_us":111,"threads_started":1}
I20260806 16:14:38.498356 31716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:14:38.503420 31716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:14:38.548732 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.529221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:37157 (local address 127.0.0.1:38192)
0806 16:14:38.536530 (+ 7309us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.538607 (+ 2077us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:38.538657 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:38.539312 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:38.539315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:38.539335 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:38.539415 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.539420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.542590 (+ 3170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.542593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.543756 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.543767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.546172 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.546175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.548229 (+ 2054us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.548303 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 16:14:38.548422 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 16:14:38.548551 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7102,"thread_start_us":118,"threads_started":1}
I20260806 16:14:38.549336 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.530104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.18.90.2:37157)
0806 16:14:38.536487 (+ 6383us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.536494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.538751 (+ 2257us) server_negotiation.cc:407] Connection header received
0806 16:14:38.538832 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.538837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.539031 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.539156 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:38.539539 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.542462 (+ 2923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.543896 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.545855 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.545954 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.546089 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.549084 (+ 2995us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.549195 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6281,"thread_start_us":91,"threads_started":1}
I20260806 16:14:38.551124 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.531667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:39345 (local address 127.0.0.1:39040)
0806 16:14:38.536587 (+ 4920us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.536623 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:38.536759 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:38.537768 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:38.537774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:38.537798 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:38.537948 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.537956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.539357 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.539361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.540767 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.540778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.550735 (+ 9957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.550739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.550826 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.550850 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:14:38.550928 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 16:14:38.550999 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4866,"thread_start_us":95,"threads_started":1}
I20260806 16:14:38.551811 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.532054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.18.90.3:39345)
0806 16:14:38.537050 (+ 4996us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.537057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.537081 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:14:38.537156 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.537162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.537405 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.537553 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:38.538081 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.539241 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.540923 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.541779 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.541889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.542033 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.551557 (+ 9524us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.551674 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4890,"thread_start_us":79,"threads_started":1}
I20260806 16:14:38.552146 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.532781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45117 (local address 127.0.0.1:57702)
0806 16:14:38.538444 (+ 5663us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.538458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:38.538540 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:38.545205 (+ 6665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:38.545207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:38.545223 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:38.545303 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.545307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.546706 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.546710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.547734 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.547744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.551803 (+ 4059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.551807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.551892 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.551914 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:14:38.551961 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 16:14:38.552023 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4715,"thread_start_us":82,"threads_started":1}
I20260806 16:14:38.553056 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.532962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.18.90.1:45117)
0806 16:14:38.544489 (+ 11527us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.544495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.544517 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:14:38.544599 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.544606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.544823 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.545007 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:38.545425 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.546581 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.550877 (+ 4296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.551644 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.552591 (+ 947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.552764 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.552822 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.552918 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11412,"thread_start_us":69,"threads_started":1}
I20260806 16:14:38.555155 32117 tablet_service.cc:1467] Processing CreateTablet for tablet 996a998f6af64b26ba9c5d241b70e82f (DEFAULT_TABLE table=test-table [id=2a1c7c8cfe2a49cf98ea04617c74dc50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:38.555159 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 996a998f6af64b26ba9c5d241b70e82f (DEFAULT_TABLE table=test-table [id=2a1c7c8cfe2a49cf98ea04617c74dc50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:38.556371 32117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996a998f6af64b26ba9c5d241b70e82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:38.556383 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996a998f6af64b26ba9c5d241b70e82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:38.560360 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 996a998f6af64b26ba9c5d241b70e82f (DEFAULT_TABLE table=test-table [id=2a1c7c8cfe2a49cf98ea04617c74dc50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:38.561431 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996a998f6af64b26ba9c5d241b70e82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:38.580374 32220 tablet_bootstrap.cc:492] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Bootstrap starting.
I20260806 16:14:38.581338 32218 tablet_bootstrap.cc:492] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: Bootstrap starting.
I20260806 16:14:38.582952 32220 tablet_bootstrap.cc:654] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:38.583590 32218 tablet_bootstrap.cc:654] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:38.584043 32220 log.cc:824] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:38.584642 32218 log.cc:824] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:38.586827 32220 tablet_bootstrap.cc:492] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: No bootstrap required, opened a new log
I20260806 16:14:38.587018 32220 ts_tablet_manager.cc:1397] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260806 16:14:38.588606 32219 tablet_bootstrap.cc:492] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: Bootstrap starting.
I20260806 16:14:38.590168 32218 tablet_bootstrap.cc:492] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: No bootstrap required, opened a new log
I20260806 16:14:38.590363 32218 ts_tablet_manager.cc:1397] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 16:14:38.590997 32219 tablet_bootstrap.cc:654] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:38.591800 32219 log.cc:824] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:38.594578 32218 raft_consensus.cc:348] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.596755 32218 raft_consensus.cc:374] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:38.596872 32218 raft_consensus.cc:729] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b883cba0a2db4bc9886972ab23d8432e, State: Initialized, Role: FOLLOWER
I20260806 16:14:38.597450 32218 consensus_queue.cc:260] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.598608 32218 ts_tablet_manager.cc:1428] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 16:14:38.599969 32220 raft_consensus.cc:348] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.600652 31915 heartbeater.cc:503] Master 127.18.90.62:44153 was elected leader, sending a full tablet report...
I20260806 16:14:38.602111 32219 tablet_bootstrap.cc:492] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: No bootstrap required, opened a new log
I20260806 16:14:38.602872 32219 ts_tablet_manager.cc:1397] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260806 16:14:38.604728 32220 raft_consensus.cc:374] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:38.604897 32220 raft_consensus.cc:729] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70eb9babc0334af18283068795deecf5, State: Initialized, Role: FOLLOWER
I20260806 16:14:38.605484 32220 consensus_queue.cc:260] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.606730 32220 ts_tablet_manager.cc:1428] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20260806 16:14:38.607125 32183 heartbeater.cc:503] Master 127.18.90.62:44153 was elected leader, sending a full tablet report...
I20260806 16:14:38.607235 32219 raft_consensus.cc:348] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.607492 32219 raft_consensus.cc:374] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:38.607540 32219 raft_consensus.cc:729] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef96ccc8d2554fbaad0830e1a344b7c2, State: Initialized, Role: FOLLOWER
I20260806 16:14:38.612170 32219 consensus_queue.cc:260] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.613516 32219 ts_tablet_manager.cc:1428] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260806 16:14:38.614012 32049 heartbeater.cc:503] Master 127.18.90.62:44153 was elected leader, sending a full tablet report...
W20260806 16:14:38.636969 31916 tablet.cc:2369] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:14:38.670691 32050 tablet.cc:2369] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:14:38.681236 32184 tablet.cc:2369] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:14:38.705037 32225 raft_consensus.cc:482] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:38.705274 32225 raft_consensus.cc:504] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.706684 32225 leader_election.cc:288] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef96ccc8d2554fbaad0830e1a344b7c2 (127.18.90.2:37157), b883cba0a2db4bc9886972ab23d8432e (127.18.90.1:45117)
I20260806 16:14:38.712952 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.707289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:40521 (local address 127.18.90.1:45117)
0806 16:14:38.707444 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.707449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.708387 (+ 938us) server_negotiation.cc:407] Connection header received
0806 16:14:38.708594 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.708597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.708649 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.708704 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:38.709146 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.709928 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.711520 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.712125 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.712205 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.712297 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.712732 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.712817 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:14:38.713660 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.707129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45117 (local address 127.18.90.3:40521)
0806 16:14:38.708294 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.708309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:38.708474 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:38.708847 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:38.708850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:38.708867 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:38.709005 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.709011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.710077 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.710080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.711373 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.711384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.712399 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.712402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.712515 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.712536 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:14:38.713427 (+ 891us) client_negotiation.cc:240] Negotiation successful
0806 16:14:38.713506 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":123,"threads_started":1}
I20260806 16:14:38.714515 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a998f6af64b26ba9c5d241b70e82f" candidate_uuid: "70eb9babc0334af18283068795deecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b883cba0a2db4bc9886972ab23d8432e" is_pre_election: true
I20260806 16:14:38.714864 31860 raft_consensus.cc:2454] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70eb9babc0334af18283068795deecf5 in term 0.
I20260806 16:14:38.715489 32071 leader_election.cc:302] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70eb9babc0334af18283068795deecf5, b883cba0a2db4bc9886972ab23d8432e; no voters:
I20260806 16:14:38.715806 32225 raft_consensus.cc:2781] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:14:38.715883 32225 raft_consensus.cc:482] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:14:38.715919 32225 raft_consensus.cc:3037] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:38.716802 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.708005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:51297 (local address 127.18.90.2:37157)
0806 16:14:38.708112 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.708116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.712351 (+ 4235us) server_negotiation.cc:407] Connection header received
0806 16:14:38.712568 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.712571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.712624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.712682 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:38.713098 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.713849 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.715426 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.716019 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.716103 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.716185 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.716579 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.716664 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:14:38.717172 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.707637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:37157 (local address 127.18.90.3:51297)
0806 16:14:38.712219 (+ 4582us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.712236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:38.712439 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:38.712825 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:38.712828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:38.712844 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:38.712952 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.712958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.714034 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.714037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.715266 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:38.715279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.716310 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.716314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:38.716416 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.716440 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:14:38.716948 (+ 508us) client_negotiation.cc:240] Negotiation successful
0806 16:14:38.717029 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4499,"thread_start_us":107,"threads_started":1}
I20260806 16:14:38.717547 32225 raft_consensus.cc:504] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.718078 32225 leader_election.cc:288] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [CANDIDATE]: Term 1 election: Requested vote from peers ef96ccc8d2554fbaad0830e1a344b7c2 (127.18.90.2:37157), b883cba0a2db4bc9886972ab23d8432e (127.18.90.1:45117)
I20260806 16:14:38.717979 31984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a998f6af64b26ba9c5d241b70e82f" candidate_uuid: "70eb9babc0334af18283068795deecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" is_pre_election: true
I20260806 16:14:38.718364 31984 raft_consensus.cc:2454] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70eb9babc0334af18283068795deecf5 in term 0.
I20260806 16:14:38.718499 31985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a998f6af64b26ba9c5d241b70e82f" candidate_uuid: "70eb9babc0334af18283068795deecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2"
I20260806 16:14:38.718597 31985 raft_consensus.cc:3037] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:38.718775 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a998f6af64b26ba9c5d241b70e82f" candidate_uuid: "70eb9babc0334af18283068795deecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b883cba0a2db4bc9886972ab23d8432e"
I20260806 16:14:38.718923 31860 raft_consensus.cc:3037] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:38.719723 31985 raft_consensus.cc:2454] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70eb9babc0334af18283068795deecf5 in term 1.
I20260806 16:14:38.720058 32073 leader_election.cc:302] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70eb9babc0334af18283068795deecf5, ef96ccc8d2554fbaad0830e1a344b7c2; no voters:
I20260806 16:14:38.720291 31860 raft_consensus.cc:2454] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70eb9babc0334af18283068795deecf5 in term 1.
I20260806 16:14:38.720259 32225 raft_consensus.cc:2781] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:38.720525 32225 raft_consensus.cc:686] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 1 LEADER]: Becoming Leader. State: Replica: 70eb9babc0334af18283068795deecf5, State: Running, Role: LEADER
I20260806 16:14:38.720902 32225 consensus_queue.cc:237] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } }
I20260806 16:14:38.724740 31717 catalog_manager.cc:5495] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70eb9babc0334af18283068795deecf5 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "70eb9babc0334af18283068795deecf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70eb9babc0334af18283068795deecf5" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 39345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 } health_report { overall_health: UNKNOWN } } }
I20260806 16:14:38.738435 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.733264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.18.90.3:39345)
0806 16:14:38.733381 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:38.733386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:38.733463 (+ 77us) server_negotiation.cc:407] Connection header received
0806 16:14:38.733660 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:38.733663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:38.733740 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:38.733847 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:14:38.734147 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.734907 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.736196 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:38.736574 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:38.736683 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:38.737005 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:14:38.737594 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:14:38.737731 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:38.738177 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 16:14:38.738260 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:14:38.746099 31860 raft_consensus.cc:1261] T 996a998f6af64b26ba9c5d241b70e82f P b883cba0a2db4bc9886972ab23d8432e [term 1 FOLLOWER]: Refusing update from remote peer 70eb9babc0334af18283068795deecf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:38.746099 31985 raft_consensus.cc:1261] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 1 FOLLOWER]: Refusing update from remote peer 70eb9babc0334af18283068795deecf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:38.747112 32229 consensus_queue.cc:1035] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b883cba0a2db4bc9886972ab23d8432e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 45117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:38.747519 32229 consensus_queue.cc:1035] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef96ccc8d2554fbaad0830e1a344b7c2" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 37157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:38.759742 32232 mvcc.cc:205] Tried to move back new op lower bound from 7315590671334367232 to 7315590671244328960. Current Snapshot: MvccSnapshot[applied={T|T < 7315590671334367232}]
I20260806 16:14:38.761442 32233 mvcc.cc:205] Tried to move back new op lower bound from 7315590671334367232 to 7315590671244328960. Current Snapshot: MvccSnapshot[applied={T|T < 7315590671334367232}]
I20260806 16:14:38.764631 32235 mvcc.cc:205] Tried to move back new op lower bound from 7315590671334367232 to 7315590671244328960. Current Snapshot: MvccSnapshot[applied={T|T < 7315590671334367232}]
I20260806 16:14:38.772143 31716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:14:38.772399 31716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:14:38.775364 31716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b539c9755419467e95c854f836c35799: Sending DeleteTablet for 3 replicas of tablet 996a998f6af64b26ba9c5d241b70e82f
I20260806 16:14:38.777226 31983 tablet_service.cc:1514] Processing DeleteTablet for tablet 996a998f6af64b26ba9c5d241b70e82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:38 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:38192
I20260806 16:14:38.777812 32117 tablet_service.cc:1514] Processing DeleteTablet for tablet 996a998f6af64b26ba9c5d241b70e82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:38 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:39040
I20260806 16:14:38.778318 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 996a998f6af64b26ba9c5d241b70e82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:38 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:57702
I20260806 16:14:38.779439 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31755
I20260806 16:14:38.782137 32246 tablet_replica.cc:330] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: stopping tablet replica
I20260806 16:14:38.782567 32246 raft_consensus.cc:2229] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:14:38.783026 32246 raft_consensus.cc:2258] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:38.783512 32244 tablet_replica.cc:330] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2: stopping tablet replica
I20260806 16:14:38.783900 32244 raft_consensus.cc:2229] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:14:38.784358 32244 raft_consensus.cc:2258] T 996a998f6af64b26ba9c5d241b70e82f P ef96ccc8d2554fbaad0830e1a344b7c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:38.786015 32246 ts_tablet_manager.cc:1905] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:14:38.788888 32246 ts_tablet_manager.cc:1918] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:14:38.788987 32246 log.cc:1196] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/996a998f6af64b26ba9c5d241b70e82f
I20260806 16:14:38.789337 32246 ts_tablet_manager.cc:1939] T 996a998f6af64b26ba9c5d241b70e82f P 70eb9babc0334af18283068795deecf5: Deleting consensus metadata
I20260806 16:14:38.790692 31701 catalog_manager.cc:4841] TS 70eb9babc0334af18283068795deecf5 (127.18.90.3:39345): tablet 996a998f6af64b26ba9c5d241b70e82f (table test-table [id=2a1c7c8cfe2a49cf98ea04617c74dc50]) successfully deleted
W20260806 16:14:38.791105 31702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:45117 (error 108)
I20260806 16:14:38.791426 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31919
I20260806 16:14:38.795519 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.795123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:45117 (local address 127.0.0.1:57710)
0806 16:14:38.795289 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.795380 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 16:14:38.795733 31702 catalog_manager.cc:4584] TS b883cba0a2db4bc9886972ab23d8432e (127.18.90.1:45117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 996a998f6af64b26ba9c5d241b70e82f: Network error: Client connection negotiation failed: client connection to 127.18.90.1:45117: connect: Connection refused (error 111)
I20260806 16:14:38.799293 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 32052
I20260806 16:14:38.799768 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:38.799404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:37157 (local address 127.0.0.1:38198)
0806 16:14:38.799596 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:38.799655 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.2:37157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 16:14:38.807312 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 31684
I20260806 16:14:38.815114 18792 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 16:14:39.161010 18792 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 16:14:39.161158 18792 mini_ranger.cc:66] Stopping Ranger...
I20260806 16:14:39.516464 18792 mini_ranger.cc:68] Stopped Ranger
2026-08-06 16:14:39.516 UTC [30743] LOG: received smart shutdown request
2026-08-06 16:14:39.521 UTC [30743] LOG: background worker "logical replication launcher" (PID 30752) exited with exit code 1
2026-08-06 16:14:39.521 UTC [30747] LOG: shutting down
2026-08-06 16:14:39.682 UTC [30743] LOG: database system is shut down
2026-08-06T16:14:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49937 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:14:39 dist-test-slave-0b58 krb5kdc[32261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:14:39 dist-test-slave-0b58 krb5kdc[32261](info): set up 2 sockets
Aug 06 16:14:39 dist-test-slave-0b58 krb5kdc[32261](info): commencing operation
krb5kdc: starting...
W20260806 16:14:40.860387 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:14:40 dist-test-slave-0b58 krb5kdc[32261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:14:40.985366 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:32873
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:32873,127.18.90.61:34941,127.18.90.60:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:41.097002 32274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:41.097316 32274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:41.097397 32274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:41.101804 32274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:41.101903 32274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:41.101938 32274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:41.101975 32274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:41.102010 32274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:41.107421 32274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:32873,127.18.90.61:34941,127.18.90.60:36683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:41.108613 32274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:41.109768 32274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:41.118069 32282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:41.118038 32279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:41.118040 32280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:41.118412 32274 server_base.cc:1034] running on GCE node
I20260806 16:14:41.119009 32274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:41.120122 32274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:41.121280 32274 hybrid_clock.cc:648] HybridClock initialized: now 1786032881121245 us; error 48 us; skew 500 ppm
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:41.127317 32274 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:41.128341 32274 webserver.cc:460] Webserver started at http://127.18.90.62:35363/ using document root <none> and password file <none>
I20260806 16:14:41.128909 32274 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:41.128971 32274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:41.129235 32274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:41.131345 32274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "c6a5e95672104ee7846aceacd455a028"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "32d34af35b2348a5a49d896b426456bb"
server_key_iv: "1a4688d8f879f9eaaf3091841d9e5c21"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.131973 32274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "c6a5e95672104ee7846aceacd455a028"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "32d34af35b2348a5a49d896b426456bb"
server_key_iv: "1a4688d8f879f9eaaf3091841d9e5c21"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.135756 32274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.007s sys 0.000s
I20260806 16:14:41.138190 32289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:41.139418 32274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:41.139559 32274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "c6a5e95672104ee7846aceacd455a028"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "32d34af35b2348a5a49d896b426456bb"
server_key_iv: "1a4688d8f879f9eaaf3091841d9e5c21"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.139671 32274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:41.160435 32274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:41.163442 32274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:41.163663 32274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:41.171156 32274 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:32873
I20260806 16:14:41.171175 32341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:32873 every 8 connection(s)
I20260806 16:14:41.172142 32274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:41.172415 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 32274
I20260806 16:14:41.172559 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:14:41.172907 18792 external_mini_cluster.cc:1428] Setting key 18f960d97109628f8eb7a341684e7c91
I20260806 16:14:41.175063 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:41.179028 32342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.90.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.61@KRBTEST.COM" created.
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:41.194730 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.181496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.18.90.62:32873)
0806 16:14:41.181915 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.181940 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.182160 (+ 220us) server_negotiation.cc:407] Connection header received
0806 16:14:41.183183 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.183188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.183427 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.183600 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.184595 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.185437 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.186702 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.187298 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.187421 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.189805 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.189811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.189820 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.189839 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.191515 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.191998 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.192010 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.192012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.192110 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.192476 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.192479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.192480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.192841 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.192985 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.193350 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.193675 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260806 16:14:41.194730 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.181497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32873 (local address 127.0.0.1:60840)
0806 16:14:41.182014 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.182052 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.182904 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.183659 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.183664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.184051 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.184412 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.184446 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.185568 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.185575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.186572 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.186582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.187443 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.187445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.187546 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.188149 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.188164 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.189646 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.191624 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.191635 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.191645 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.191873 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.192205 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.192209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.192214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.192380 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.192956 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.192963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.193118 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.193279 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.193730 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":163,"spinlock_wait_cycles":1792,"thread_start_us":172,"threads_started":1}
I20260806 16:14:41.197256 32342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:14:41.198040 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.197761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34334)
0806 16:14:41.197901 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.197972 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 16:14:41.198258 32291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
I20260806 16:14:41.199177 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.198952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34336)
0806 16:14:41.199075 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.199127 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 16:14:41.199339 32342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:34941: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab.
Entry for principal kudu/127.18.90.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab.
I20260806 16:14:41.202765 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:34941
--webserver_interface=127.18.90.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab
--principal=kudu/127.18.90.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:32873,127.18.90.61:34941,127.18.90.60:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:41.248605 32342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } attempt: 1
I20260806 16:14:41.249543 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.249238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34342)
0806 16:14:41.249398 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.249463 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 16:14:41.250291 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.250079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34356)
0806 16:14:41.250185 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.250236 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 16:14:41.250422 32342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:34941: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
W20260806 16:14:41.317225 32349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:41.317517 32349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:41.317597 32349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:14:41.318594 32342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } attempt: 2
I20260806 16:14:41.319435 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.319137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34360)
0806 16:14:41.319290 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.319360 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 16:14:41.320158 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.319955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34362)
0806 16:14:41.320056 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.320101 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:14:41.320333 32342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:34941: Network error: Client connection negotiation failed: client connection to 127.18.90.61:34941: connect: Connection refused (error 111)
W20260806 16:14:41.321890 32349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:41.321942 32349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:41.321983 32349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:41.322037 32349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:41.322088 32349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:41.327487 32349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:32873,127.18.90.61:34941,127.18.90.60:36683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab
--principal=kudu/127.18.90.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.90.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:41.328722 32349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:41.329787 32349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:41.338307 32354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:41.338338 32355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:41.338610 32349 server_base.cc:1034] running on GCE node
W20260806 16:14:41.338655 32357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:41.339380 32349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:41.340597 32349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:41.341768 32349 hybrid_clock.cc:648] HybridClock initialized: now 1786032881341734 us; error 47 us; skew 500 ppm
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:41.344605 32349 init.cc:377] Logged in from keytab as kudu/127.18.90.61@KRBTEST.COM (short username kudu)
I20260806 16:14:41.345628 32349 webserver.cc:460] Webserver started at http://127.18.90.61:44815/ using document root <none> and password file <none>
I20260806 16:14:41.346153 32349 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:41.346233 32349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:41.346400 32349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:41.348469 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data/instance:
uuid: "f6456ea5645148c78fd5c2a48b8131f2"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "d19753267ec90c239daf0001a13e8326"
server_key_iv: "8df1c66d0178e2a429559ce7b2ceda65"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.348994 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal/instance:
uuid: "f6456ea5645148c78fd5c2a48b8131f2"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "d19753267ec90c239daf0001a13e8326"
server_key_iv: "8df1c66d0178e2a429559ce7b2ceda65"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.352720 32349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:14:41.355047 32364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:41.356247 32349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:41.356406 32349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal
uuid: "f6456ea5645148c78fd5c2a48b8131f2"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "d19753267ec90c239daf0001a13e8326"
server_key_iv: "8df1c66d0178e2a429559ce7b2ceda65"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.356491 32349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:41.373494 32349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:41.376284 32349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:41.376434 32349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:41.383553 32349 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.61:34941
I20260806 16:14:41.383580 32416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.61:34941 every 8 connection(s)
I20260806 16:14:41.384511 32349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
I20260806 16:14:41.387187 32417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:41.390026 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 32349
I20260806 16:14:41.390129 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-1/wal/instance
I20260806 16:14:41.390425 18792 external_mini_cluster.cc:1428] Setting key fbbd790c54e32609b7852a2b8b14a90c
I20260806 16:14:41.391142 32417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.90.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.60@KRBTEST.COM" created.
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:41.406611 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.393092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.18.90.62:32873)
0806 16:14:41.393232 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.393236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.394226 (+ 990us) server_negotiation.cc:407] Connection header received
0806 16:14:41.395127 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.395131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.395198 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.395290 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.396481 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.396999 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.398091 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.398330 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.398438 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.402571 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.402577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.402583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.402621 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.404160 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.404855 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.404857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.404859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.404933 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.405340 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.405342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.405343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.405515 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.405585 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.406370 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.406484 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:14:41.407909 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.393580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32873 (local address 127.0.0.1:60848)
0806 16:14:41.394074 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.394115 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.394913 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.395466 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.395472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.395800 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.396326 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.396349 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.397137 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.397141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.397945 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.397953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.398479 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.398483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.398673 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.399328 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.399344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.402381 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.404329 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.404339 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.404358 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.404748 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.405020 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.405026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.405030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.405243 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.405615 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.405624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.406005 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.406271 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.406673 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":144,"threads_started":1}
I20260806 16:14:41.409492 32417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:14:41.411492 32342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } attempt: 3
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab.
Entry for principal kudu/127.18.90.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab.
I20260806 16:14:41.418869 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:36683
--webserver_interface=127.18.90.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab
--principal=kudu/127.18.90.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:32873,127.18.90.61:34941,127.18.90.60:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:41.420138 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.410081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34378)
0806 16:14:41.410294 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.410311 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.410401 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.410909 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.410912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.411090 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.411181 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.411186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.412639 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.412641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.413245 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.413276 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.414082 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.414088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.414202 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.414692 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.414706 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.416088 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.418347 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.418349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.418350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.418603 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.418972 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.418976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.418979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.419064 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.419623 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.419626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.419710 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.419833 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.419963 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 16:14:41.420390 32422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.410189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.18.90.61:34941)
0806 16:14:41.410560 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.410569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.410588 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:14:41.410622 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.410626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.410772 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.410882 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.411296 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.412478 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.413408 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.413891 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.414011 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.416205 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.416215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.416238 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.416335 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.418232 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.418739 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.418747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.418749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.418870 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.419169 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.419172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.419174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.419499 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.419647 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.419935 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.420060 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":157,"threads_started":1}
I20260806 16:14:41.422433 32417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:14:41.422598 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.411858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34386)
0806 16:14:41.412015 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.412027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.412094 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.412500 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.412503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.412681 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.412788 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.412794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.413646 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.413648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.414958 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.414970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.415603 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.415606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.415717 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.416329 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.416348 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.417819 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.421075 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.421078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.421080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.421336 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.421609 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.421612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.421614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.421699 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.422067 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.422070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.422189 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.422293 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.422425 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 16:14:41.423194 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.422926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41316)
0806 16:14:41.423058 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.423134 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 16:14:41.423211 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.411933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.18.90.61:34941)
0806 16:14:41.412240 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.412246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.412302 (+ 56us) server_negotiation.cc:407] Connection header received
0806 16:14:41.412334 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.412336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.412377 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.412460 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.412906 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.413544 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.415116 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.415478 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.415599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.417947 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.417950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.417953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.417982 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.420956 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.421447 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.421449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.421450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.421515 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.421804 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.421808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.421809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.421970 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.422074 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.422928 (+ 854us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.423059 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
W20260806 16:14:41.423418 32368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
I20260806 16:14:41.423797 32342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:14:41.424531 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.424223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41332)
0806 16:14:41.424386 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.424446 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 16:14:41.424557 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.424213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41326)
0806 16:14:41.424460 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.424500 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260806 16:14:41.424763 32417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:36683: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
I20260806 16:14:41.425261 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.425011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41340)
0806 16:14:41.425155 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.425201 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 16:14:41.425400 32342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:36683: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
I20260806 16:14:41.456658 32342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } attempt: 1
I20260806 16:14:41.457619 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.457250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41356)
0806 16:14:41.457440 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.457518 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 16:14:41.458360 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.458133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41364)
0806 16:14:41.458244 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.458297 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 16:14:41.458506 32342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:36683: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
I20260806 16:14:41.474116 32417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } attempt: 1
I20260806 16:14:41.475039 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.474737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41374)
0806 16:14:41.474895 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.474962 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 16:14:41.475808 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.475611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41382)
0806 16:14:41.475709 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.475758 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 16:14:41.475977 32417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:36683: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
I20260806 16:14:41.533701 32342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } attempt: 2
I20260806 16:14:41.534545 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.534225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41386)
0806 16:14:41.534384 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.534455 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 16:14:41.535343 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.535111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41396)
0806 16:14:41.535241 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.535291 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 16:14:41.535482 32342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:36683: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
W20260806 16:14:41.536806 32424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:41.537088 32424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:41.537194 32424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:41.541257 32424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:41.541337 32424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:41.541397 32424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:41.541455 32424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:41.541491 32424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:41.544116 32417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } attempt: 2
I20260806 16:14:41.544872 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.544591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41402)
0806 16:14:41.544734 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.544791 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 16:14:41.545678 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.545405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41404)
0806 16:14:41.545535 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.545591 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 16:14:41.545805 32417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:36683: Network error: Client connection negotiation failed: client connection to 127.18.90.60:36683: connect: Connection refused (error 111)
I20260806 16:14:41.547207 32424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:32873,127.18.90.61:34941,127.18.90.60:36683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab
--principal=kudu/127.18.90.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.90.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:41.548363 32424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:41.549428 32424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:41.558327 32431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:41.558327 32433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:41.558398 32424 server_base.cc:1034] running on GCE node
W20260806 16:14:41.558341 32430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:41.559103 32424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:41.560421 32424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:41.561604 32424 hybrid_clock.cc:648] HybridClock initialized: now 1786032881561570 us; error 47 us; skew 500 ppm
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:41.564394 32424 init.cc:377] Logged in from keytab as kudu/127.18.90.60@KRBTEST.COM (short username kudu)
I20260806 16:14:41.565485 32424 webserver.cc:460] Webserver started at http://127.18.90.60:41433/ using document root <none> and password file <none>
I20260806 16:14:41.566128 32424 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:41.566251 32424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:41.566545 32424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:41.568722 32424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data/instance:
uuid: "990f38f0da264d9aba51d1edb2cef9e3"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "32588367aa0f95a8a13eb796fea0dada"
server_key_iv: "c53d357aef1fd15bfd7dff78602b316c"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.569300 32424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal/instance:
uuid: "990f38f0da264d9aba51d1edb2cef9e3"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "32588367aa0f95a8a13eb796fea0dada"
server_key_iv: "c53d357aef1fd15bfd7dff78602b316c"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.573109 32424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:14:41.575589 32440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:41.576889 32424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:14:41.577039 32424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal
uuid: "990f38f0da264d9aba51d1edb2cef9e3"
format_stamp: "Formatted at 2026-08-06 16:14:41 on dist-test-slave-0b58"
server_key: "32588367aa0f95a8a13eb796fea0dada"
server_key_iv: "c53d357aef1fd15bfd7dff78602b316c"
server_key_version: "encryptionkey@0"
I20260806 16:14:41.577157 32424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:41.589075 32424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:41.591893 32424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:41.592100 32424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:41.598851 32424 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.60:36683
I20260806 16:14:41.598860 32492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.60:36683 every 8 connection(s)
I20260806 16:14:41.599852 32424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
I20260806 16:14:41.602614 32493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:41.606379 32493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:14:41.607391 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 32424
I20260806 16:14:41.607513 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032720994684-18792-0/minicluster-data/master-2/wal/instance
I20260806 16:14:41.607800 18792 external_mini_cluster.cc:1428] Setting key 1872a94d8025bf828b149dbcd48af0f0
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:14:41.620091 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.608547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.18.90.62:32873)
0806 16:14:41.608680 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.608684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.609591 (+ 907us) server_negotiation.cc:407] Connection header received
0806 16:14:41.610441 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.610445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.610509 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.610608 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.611569 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.612058 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.613160 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.613432 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.613511 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.616240 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.616242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.616245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.616309 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.617824 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.618335 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.618338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.618339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.618413 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.618771 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.618774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.618776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.619046 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.619119 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.619770 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.619902 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:14:41.621047 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.608961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.18.90.60:36683)
0806 16:14:41.609437 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.609455 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.609495 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:14:41.610226 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.610244 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.610510 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.610713 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.611429 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.612437 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.613263 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.613830 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.613925 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.616686 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.616695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.616709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.616735 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.618554 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.619024 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.619028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.619029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.619119 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.619400 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.619403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.619404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.619654 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.619834 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.620073 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.620484 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":359,"spinlock_wait_cycles":15616,"thread_start_us":208,"threads_started":1}
I20260806 16:14:41.621047 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.608961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:32873 (local address 127.0.0.1:60864)
0806 16:14:41.609455 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.609492 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.610238 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.610694 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.610699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.611135 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.611419 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.611431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.612169 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.612172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.613039 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.613048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.613570 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.613574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.613888 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.614520 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.614535 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.616047 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.617931 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.617938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.617949 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.618238 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.618503 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.618508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.618511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.618676 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.619140 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.619149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.619475 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.619680 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.620019 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":78,"spinlock_wait_cycles":1792,"thread_start_us":119,"threads_started":1}
I20260806 16:14:41.622476 32493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:41.631305 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.622961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:34941 (local address 127.0.0.1:34392)
0806 16:14:41.623065 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.623079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.623148 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.623568 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.623570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.623695 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.623785 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.623789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.624785 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.624788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.625424 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.625432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.626016 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.626019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.626115 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.626608 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.626619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.627843 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.629745 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.629755 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.629758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.629992 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.630257 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.630264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.630267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.630371 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.630802 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.630804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.630900 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.631003 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.631117 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 16:14:41.631323 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.623108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.18.90.61:34941)
0806 16:14:41.623262 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.623267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.623281 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:14:41.623325 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.623328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.623380 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.623458 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.623883 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.624679 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.625535 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.625910 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.626010 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.627988 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.627991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.627993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.628016 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.629638 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.630091 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.630093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.630094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.630166 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.630465 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.630468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.630473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.630704 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.630798 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.631069 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.631200 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:14:41.632063 32493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:14:41.632336 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.623378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.18.90.62:32873)
0806 16:14:41.623482 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.623488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.623501 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:14:41.623608 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.623611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.623646 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.623736 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.624447 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.625039 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.625932 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.626281 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.626378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.628747 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.628751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.628754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.628776 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.630311 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.630802 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.630805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.630807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.630890 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.631192 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.631195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.631197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.631417 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.631633 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.631996 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.632111 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:14:41.634639 32342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } attempt: 3
I20260806 16:14:41.636969 32417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } attempt: 3
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:14:41.641734 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.632602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41414)
0806 16:14:41.632693 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.632708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.632790 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.633093 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.633096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.633215 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.633330 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.633340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.634268 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.634274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.635320 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.635329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.635873 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.635877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.635994 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.636488 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.636504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.637869 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.639811 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.639818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.639821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.640043 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.640645 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.640648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.640651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.640746 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.641190 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.641195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.641301 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.641396 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.641533 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":18}
I20260806 16:14:41.642617 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.632671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.18.90.60:36683)
0806 16:14:41.632779 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.632783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.632797 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:14:41.632850 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.632852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.632897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.632975 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.633454 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.634164 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.635433 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.635771 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.635871 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.637998 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.638002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.638006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.638030 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.639694 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.640158 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.640162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.640164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.640547 (+ 383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.640860 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.640864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.640866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.641085 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.642243 (+ 1158us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.642321 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.642434 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":43}
Aug 06 16:14:41 dist-test-slave-0b58 krb5kdc[32261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:14:41.644984 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.633532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.18.90.61:34941)
0806 16:14:41.633647 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.633650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.633667 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:14:41.633759 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.633761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.633816 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.633893 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.634582 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.635171 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.636378 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.636762 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.636836 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.640446 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.640452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.640454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.640479 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.642653 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.643462 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.643468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.643469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.643550 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.643908 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.643911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.643913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.644077 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.644370 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.644782 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.644880 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:14:41.645903 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.634972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41416)
0806 16:14:41.635099 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.635112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.635182 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.635729 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.635732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.635910 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.635997 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.636004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.638410 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.638416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.639225 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.639238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.639861 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.639865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.639948 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.640412 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.640430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.641757 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.644205 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.644208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.644226 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.644490 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.644843 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.644846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.644848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.644930 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.645404 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.645407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.645507 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.645611 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.645774 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 16:14:41.646361 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.635080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.18.90.60:36683)
0806 16:14:41.635548 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.635554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.635562 (+ 8us) server_negotiation.cc:407] Connection header received
0806 16:14:41.635586 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.635587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.635616 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.635711 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.636132 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.638267 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.639384 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.639734 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.640149 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.641900 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.641906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.641912 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.641941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.644095 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.644642 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.644648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.644650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.644732 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.645032 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.645035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.645037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.645275 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.645404 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.646118 (+ 714us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.646246 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"spinlock_wait_cycles":4352,"thread_start_us":145,"threads_started":1}
I20260806 16:14:41.647030 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3: Bootstrap starting.
I20260806 16:14:41.649147 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:14:41.649537 32342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: Bootstrap starting.
I20260806 16:14:41.649960 32493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:41.650868 32493 log.cc:824] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:41.652228 32342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:41.653172 32342 log.cc:824] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:41.655015 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.637340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:36683 (local address 127.0.0.1:41420)
0806 16:14:41.637461 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:41.637475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:41.637544 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:41.639111 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:41.639114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:41.639301 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:41.639400 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.639407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.640542 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.640546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.641363 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:41.641375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.641990 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.641996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:41.642093 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.642597 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:41.642614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:41.643994 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:41.653354 (+ 9360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.653358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.653360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.653644 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.654004 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:41.654010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:41.654012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:41.654102 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:41.654525 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:41.654528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:41.654627 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 16:14:41.654715 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 16:14:41.654864 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 16:14:41.655376 32342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: No bootstrap required, opened a new log
I20260806 16:14:41.655638 32499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.637612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.18.90.60:36683)
0806 16:14:41.638894 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.638901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.638914 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:14:41.638945 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.638947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.638982 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.639493 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.639609 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.640396 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.641519 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.641872 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.642271 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.651335 (+ 9064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.651339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.651341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.651363 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.653205 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.653795 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.653798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.653800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.653884 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.654213 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.654216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.654217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.654411 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.655294 (+ 883us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.655370 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.655491 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":111,"threads_started":1}
I20260806 16:14:41.657333 32493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3: No bootstrap required, opened a new log
I20260806 16:14:41.659515 32342 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.659761 32342 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:41.659819 32342 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a5e95672104ee7846aceacd455a028, State: Initialized, Role: FOLLOWER
I20260806 16:14:41.662200 32493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.662448 32493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:41.662504 32493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990f38f0da264d9aba51d1edb2cef9e3, State: Initialized, Role: FOLLOWER
I20260806 16:14:41.662789 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.652034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.18.90.60:36683)
0806 16:14:41.652135 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.652141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.652770 (+ 629us) server_negotiation.cc:407] Connection header received
0806 16:14:41.652898 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.652900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.652941 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.653017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.653827 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.654576 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.655867 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.656191 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.656302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.658135 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.658139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.658141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.658166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.660139 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.660745 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.660750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.660754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.660872 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.661210 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.661216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.661218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.661481 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.661647 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.662356 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.662637 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 16:14:41.662972 32493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.663200 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: Bootstrap starting.
I20260806 16:14:41.663739 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.651294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.18.90.62:32873)
0806 16:14:41.651412 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.651417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.652189 (+ 772us) server_negotiation.cc:407] Connection header received
0806 16:14:41.652258 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.652285 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.652416 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.652494 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.653827 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.654576 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.656343 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.656653 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.656757 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.659750 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.659756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.659760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.659789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.661648 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.662094 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.662098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.662101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.662196 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.662513 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.662518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.662520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.662744 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.662860 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.663494 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.663627 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:14:41.664402 32493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:41.665336 32508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.665474 32508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.667824 32417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:41.668824 32417 log.cc:824] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:41.669566 32342 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
W20260806 16:14:41.670771 32519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:14:41.670889 32519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:14:41.671025 32342 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:41.668166 32424 master_runner.cc:186] Error getting master registration for 127.18.90.62:32873: OK, instance_id { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" instance_seqno: 1786032881168987 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:14:41.671545 32424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:14:41.674440 32417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: No bootstrap required, opened a new log
I20260806 16:14:41.674567 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:41.664642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.18.90.61:34941)
0806 16:14:41.664780 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:41.664785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:41.665228 (+ 443us) server_negotiation.cc:407] Connection header received
0806 16:14:41.665437 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:41.665440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:41.665505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:41.665564 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:41.666436 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.667186 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.668701 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:41.669028 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:41.669124 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:41.670894 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:41.670901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:41.670906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:41.670936 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:41.672717 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.673221 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.673224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.673226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.673317 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:41.673620 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:41.673623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:41.673625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:41.673854 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:41.673982 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:41.674264 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 16:14:41.674381 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:14:41.677731 32417 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.677968 32417 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:41.678027 32417 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6456ea5645148c78fd5c2a48b8131f2, State: Initialized, Role: FOLLOWER
I20260806 16:14:41.678442 32417 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.678921 32532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.679934 32532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.680011 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.680234 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.681610 32417 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:14:41.683948 32534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:14:41.684053 32534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 16:14:41.687681 32546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:14:41.687815 32546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:14:41.683394 32274 master_runner.cc:186] Error getting master registration for 127.18.90.61:34941: OK, instance_id { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" instance_seqno: 1786032881381504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:14:41.687994 32274 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:14:41.691527 32349 master_runner.cc:263] No leader master found from master f6456ea5645148c78fd5c2a48b8131f2
I20260806 16:14:41.691609 32349 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:14:41.746807 32532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:41.747025 32532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.748183 32532 leader_election.cc:288] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a5e95672104ee7846aceacd455a028 (127.18.90.62:32873), 990f38f0da264d9aba51d1edb2cef9e3 (127.18.90.60:36683)
I20260806 16:14:41.748749 32508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:41.748922 32508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.749884 32508 leader_election.cc:288] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a5e95672104ee7846aceacd455a028 (127.18.90.62:32873), f6456ea5645148c78fd5c2a48b8131f2 (127.18.90.61:34941)
I20260806 16:14:41.750694 32316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990f38f0da264d9aba51d1edb2cef9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a5e95672104ee7846aceacd455a028" is_pre_election: true
I20260806 16:14:41.751024 32316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990f38f0da264d9aba51d1edb2cef9e3 in term 0.
I20260806 16:14:41.751494 32441 leader_election.cc:302] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 990f38f0da264d9aba51d1edb2cef9e3, c6a5e95672104ee7846aceacd455a028; no voters:
I20260806 16:14:41.752135 32391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990f38f0da264d9aba51d1edb2cef9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6456ea5645148c78fd5c2a48b8131f2" is_pre_election: true
I20260806 16:14:41.752391 32391 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990f38f0da264d9aba51d1edb2cef9e3 in term 0.
I20260806 16:14:41.752992 32508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:14:41.753087 32508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:14:41.753158 32508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:41.754843 32508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.755851 32508 leader_election.cc:288] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [CANDIDATE]: Term 1 election: Requested vote from peers c6a5e95672104ee7846aceacd455a028 (127.18.90.62:32873), f6456ea5645148c78fd5c2a48b8131f2 (127.18.90.61:34941)
I20260806 16:14:41.756462 32391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990f38f0da264d9aba51d1edb2cef9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6456ea5645148c78fd5c2a48b8131f2"
I20260806 16:14:41.756592 32391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:41.757897 32391 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990f38f0da264d9aba51d1edb2cef9e3 in term 1.
I20260806 16:14:41.758226 32442 leader_election.cc:302] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 990f38f0da264d9aba51d1edb2cef9e3, f6456ea5645148c78fd5c2a48b8131f2; no voters:
I20260806 16:14:41.758472 32316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "990f38f0da264d9aba51d1edb2cef9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a5e95672104ee7846aceacd455a028"
I20260806 16:14:41.758633 32316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:41.759946 32316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990f38f0da264d9aba51d1edb2cef9e3 in term 1.
I20260806 16:14:41.760923 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6456ea5645148c78fd5c2a48b8131f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "990f38f0da264d9aba51d1edb2cef9e3" is_pre_election: true
I20260806 16:14:41.761176 32467 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6456ea5645148c78fd5c2a48b8131f2 in current term 1: Already voted for candidate 990f38f0da264d9aba51d1edb2cef9e3 in this term.
I20260806 16:14:41.764314 32316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6456ea5645148c78fd5c2a48b8131f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a5e95672104ee7846aceacd455a028" is_pre_election: true
I20260806 16:14:41.764503 32316 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6456ea5645148c78fd5c2a48b8131f2 in current term 1: Already voted for candidate 990f38f0da264d9aba51d1edb2cef9e3 in this term.
I20260806 16:14:41.764904 32365 leader_election.cc:302] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6456ea5645148c78fd5c2a48b8131f2; no voters: 990f38f0da264d9aba51d1edb2cef9e3, c6a5e95672104ee7846aceacd455a028
I20260806 16:14:41.765391 32532 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 16:14:41.765576 32508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:41.765815 32508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [term 1 LEADER]: Becoming Leader. State: Replica: 990f38f0da264d9aba51d1edb2cef9e3, State: Running, Role: LEADER
I20260806 16:14:41.766203 32508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } }
I20260806 16:14:41.768043 32508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990f38f0da264d9aba51d1edb2cef9e3. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.768154 32508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:41.770035 32550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:41.771023 32550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:41.776930 32391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [term 1 FOLLOWER]: Refusing update from remote peer 990f38f0da264d9aba51d1edb2cef9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:41.779994 32548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:41.784650 32532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990f38f0da264d9aba51d1edb2cef9e3. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.784789 32532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.785768 32316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [term 1 FOLLOWER]: Refusing update from remote peer 990f38f0da264d9aba51d1edb2cef9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:41.786352 32548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:41.789965 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990f38f0da264d9aba51d1edb2cef9e3. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.790099 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.800230 32548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.800438 32548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:41.801847 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.801987 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.802393 32551 mvcc.cc:205] Tried to move back new op lower bound from 7315590683749396480 to 7315590683717718016. Current Snapshot: MvccSnapshot[applied={T|T < 7315590683749396480}]
I20260806 16:14:41.803416 32508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.803539 32508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3 [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:41.804677 32532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "990f38f0da264d9aba51d1edb2cef9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a5e95672104ee7846aceacd455a028" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 32873 } } peers { permanent_uuid: "f6456ea5645148c78fd5c2a48b8131f2" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 34941 } } peers { permanent_uuid: "990f38f0da264d9aba51d1edb2cef9e3" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 36683 } } }
I20260806 16:14:41.804817 32532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:14:41.805553 32554 mvcc.cc:205] Tried to move back new op lower bound from 7315590683749396480 to 7315590683717718016. Current Snapshot: MvccSnapshot[applied={T|T < 7315590683749396480}]
I20260806 16:14:41.805737 32552 mvcc.cc:205] Tried to move back new op lower bound from 7315590683749396480 to 7315590683717718016. Current Snapshot: MvccSnapshot[applied={T|T < 7315590683749396480}]
I20260806 16:14:41.813977 32550 catalog_manager.cc:1347] Generated new cluster ID: 5151a43953c2499fb255462506de32f3
I20260806 16:14:41.814067 32550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:14:41.836647 32550 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:14:41.837993 32550 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:14:41.866743 32550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 990f38f0da264d9aba51d1edb2cef9e3: Generated new TSK 0
I20260806 16:14:41.867751 32550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:14:42.685346 32534 catalog_manager.cc:1259] Loaded cluster ID: 5151a43953c2499fb255462506de32f3
I20260806 16:14:42.685456 32534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: loading cluster ID for follower catalog manager: success
I20260806 16:14:42.687099 32534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: acquiring CA information for follower catalog manager: success
I20260806 16:14:42.688352 32534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6a5e95672104ee7846aceacd455a028: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:14:42.688992 32546 catalog_manager.cc:1259] Loaded cluster ID: 5151a43953c2499fb255462506de32f3
I20260806 16:14:42.689054 32546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: loading cluster ID for follower catalog manager: success
I20260806 16:14:42.690531 32546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: acquiring CA information for follower catalog manager: success
I20260806 16:14:42.691573 32546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6456ea5645148c78fd5c2a48b8131f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:14:42.939443 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 32274
I20260806 16:14:42.946772 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 32349
I20260806 16:14:42.953519 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 32424
2026-08-06T16:14:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3142 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:14:43 dist-test-slave-0b58 krb5kdc[32564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:14:43 dist-test-slave-0b58 krb5kdc[32564](info): set up 2 sockets
Aug 06 16:14:43 dist-test-slave-0b58 krb5kdc[32564](info): commencing operation
krb5kdc: starting...
W20260806 16:14:43.972553 18792 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:14:44 dist-test-slave-0b58 krb5kdc[32564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:14:44.100356 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:43645
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:44.212805 32577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:44.213058 32577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:44.213096 32577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:44.217044 32577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:44.217096 32577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:44.217115 32577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:44.217132 32577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:44.217149 32577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:44.222128 32577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:44.223171 32577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:44.224170 32577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:44.233719 32585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:44.233693 32582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:44.233814 32577 server_base.cc:1034] running on GCE node
W20260806 16:14:44.233705 32583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:44.234558 32577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:44.235630 32577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:44.236830 32577 hybrid_clock.cc:648] HybridClock initialized: now 1786032884236817 us; error 31 us; skew 500 ppm
Aug 06 16:14:44 dist-test-slave-0b58 krb5kdc[32564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032884, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:44.239434 32577 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:44.240381 32577 webserver.cc:460] Webserver started at http://127.18.90.62:39881/ using document root <none> and password file <none>
I20260806 16:14:44.240953 32577 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:44.241044 32577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:44.241258 32577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:44.243256 32577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "b76af891d8684fc3a86e8f5474e991bc"
format_stamp: "Formatted at 2026-08-06 16:14:44 on dist-test-slave-0b58"
server_key: "3a216b27f4cec11e7ab2d785aa0521ca"
server_key_iv: "4b62f38ba5630a8b12cfc8311a2d89c7"
server_key_version: "encryptionkey@0"
I20260806 16:14:44.243825 32577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "b76af891d8684fc3a86e8f5474e991bc"
format_stamp: "Formatted at 2026-08-06 16:14:44 on dist-test-slave-0b58"
server_key: "3a216b27f4cec11e7ab2d785aa0521ca"
server_key_iv: "4b62f38ba5630a8b12cfc8311a2d89c7"
server_key_version: "encryptionkey@0"
I20260806 16:14:44.247444 32577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:14:44.249892 32592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:44.251053 32577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:14:44.251195 32577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "b76af891d8684fc3a86e8f5474e991bc"
format_stamp: "Formatted at 2026-08-06 16:14:44 on dist-test-slave-0b58"
server_key: "3a216b27f4cec11e7ab2d785aa0521ca"
server_key_iv: "4b62f38ba5630a8b12cfc8311a2d89c7"
server_key_version: "encryptionkey@0"
I20260806 16:14:44.251313 32577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:44.267179 32577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:44.269961 32577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:44.270171 32577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:44.277169 32577 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:43645
I20260806 16:14:44.277196 32644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:43645 every 8 connection(s)
I20260806 16:14:44.278162 32577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:44.280910 32645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:44.286309 32645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc: Bootstrap starting.
I20260806 16:14:44.287377 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 32577
I20260806 16:14:44.287515 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:14:44.287824 18792 external_mini_cluster.cc:1428] Setting key 100b410ddee4eb345098fdaf802f0be0
I20260806 16:14:44.289294 32645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:44.290020 32645 log.cc:824] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:44.292214 32645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc: No bootstrap required, opened a new log
I20260806 16:14:44.296759 32645 raft_consensus.cc:348] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76af891d8684fc3a86e8f5474e991bc" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43645 } }
Aug 06 16:14:44 dist-test-slave-0b58 krb5kdc[32564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:44.297020 32645 raft_consensus.cc:374] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:44.297120 32645 raft_consensus.cc:729] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76af891d8684fc3a86e8f5474e991bc, State: Initialized, Role: FOLLOWER
I20260806 16:14:44.297639 32645 consensus_queue.cc:260] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76af891d8684fc3a86e8f5474e991bc" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43645 } }
I20260806 16:14:44.297784 32645 raft_consensus.cc:388] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:14:44.297871 32645 raft_consensus.cc:482] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:14:44.297978 32645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:44.299074 32645 raft_consensus.cc:504] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76af891d8684fc3a86e8f5474e991bc" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43645 } }
I20260806 16:14:44.299435 32645 leader_election.cc:302] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b76af891d8684fc3a86e8f5474e991bc; no voters:
I20260806 16:14:44.299744 32645 leader_election.cc:288] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:14:44.299868 32650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:44.300138 32650 raft_consensus.cc:686] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [term 1 LEADER]: Becoming Leader. State: Replica: b76af891d8684fc3a86e8f5474e991bc, State: Running, Role: LEADER
I20260806 16:14:44.300516 32650 consensus_queue.cc:237] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76af891d8684fc3a86e8f5474e991bc" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43645 } }
I20260806 16:14:44.301141 32645 sys_catalog.cc:564] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:44.301733 32652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b76af891d8684fc3a86e8f5474e991bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76af891d8684fc3a86e8f5474e991bc" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43645 } } }
I20260806 16:14:44.301879 32652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:44.302305 32656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:44.302186 32651 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b76af891d8684fc3a86e8f5474e991bc. Latest consensus state: current_term: 1 leader_uuid: "b76af891d8684fc3a86e8f5474e991bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76af891d8684fc3a86e8f5474e991bc" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43645 } } }
I20260806 16:14:44.302383 32651 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:44.303683 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:44.288929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.18.90.62:43645)
0806 16:14:44.289354 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:44.289369 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:44.289429 (+ 60us) server_negotiation.cc:407] Connection header received
0806 16:14:44.290085 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:44.290101 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:44.290551 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:44.290920 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:44.291775 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:44.292860 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:44.294052 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:44.294524 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:44.294641 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:44.297668 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:44.297673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:44.297686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:44.297713 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:44.299685 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:44.300282 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:44.300289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:44.300295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:44.300440 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:44.300821 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:44.300825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:44.300827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:44.301341 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:44.301556 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:44.302399 (+ 843us) server_negotiation.cc:299] Negotiation successful
0806 16:14:44.302691 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260806 16:14:44.306407 32656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:44.306764 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:14:44.312417 32656 catalog_manager.cc:1347] Generated new cluster ID: 648fd863e7fa447bac284516cdbec6dd
I20260806 16:14:44.312487 32656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:14:44.320920 32656 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:14:44.321816 32656 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:14:44.332759 32656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b76af891d8684fc3a86e8f5474e991bc: Generated new TSK 0
I20260806 16:14:44.333495 32656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:14:44.363490 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 32577
2026-08-06T16:14:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1408 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:14:44 dist-test-slave-0b58 krb5kdc[32672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:14:44 dist-test-slave-0b58 krb5kdc[32672](info): set up 2 sockets
Aug 06 16:14:44 dist-test-slave-0b58 krb5kdc[32672](info): commencing operation
krb5kdc: starting...
W20260806 16:14:45.388196 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:14:45 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:14:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:14:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:14:45.511313 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38827
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:45.622419 32685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:45.622692 32685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:45.622763 32685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:45.626710 32685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:45.626804 32685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:45.626843 32685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:45.626896 32685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:45.626935 32685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:45.632040 32685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:38827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:45.633164 32685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:45.634220 32685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:45.642561 32693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:45.642552 32691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:45.642536 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:45.642789 32685 server_base.cc:1034] running on GCE node
I20260806 16:14:45.643491 32685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:45.644696 32685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:45.645859 32685 hybrid_clock.cc:648] HybridClock initialized: now 1786032885645827 us; error 45 us; skew 500 ppm
Aug 06 16:14:45 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:45.648722 32685 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:45.649590 32685 webserver.cc:460] Webserver started at http://127.18.90.62:44499/ using document root <none> and password file <none>
I20260806 16:14:45.650137 32685 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:45.650230 32685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:45.650475 32685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:45.652549 32685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "d4763667d0c346918de89d738fa682dd"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "ca06b48a362b76b0b4933e843482c111"
server_key_iv: "c249f4345ff80707ab98f3e1c723cf9e"
server_key_version: "encryptionkey@0"
I20260806 16:14:45.653117 32685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "d4763667d0c346918de89d738fa682dd"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "ca06b48a362b76b0b4933e843482c111"
server_key_iv: "c249f4345ff80707ab98f3e1c723cf9e"
server_key_version: "encryptionkey@0"
I20260806 16:14:45.656663 32685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:14:45.658938 32700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:45.660110 32685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:45.660248 32685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "d4763667d0c346918de89d738fa682dd"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "ca06b48a362b76b0b4933e843482c111"
server_key_iv: "c249f4345ff80707ab98f3e1c723cf9e"
server_key_version: "encryptionkey@0"
I20260806 16:14:45.660426 32685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:45.673036 32685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:45.675661 32685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:45.675851 32685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:45.682603 32685 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:38827
I20260806 16:14:45.682879 32753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:38827 every 8 connection(s)
I20260806 16:14:45.683566 32685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:45.686427 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:45.687585 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 32685
I20260806 16:14:45.687687 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:14:45.688014 18792 external_mini_cluster.cc:1428] Setting key e02c9ea01c015c9a9eb914ae1ea8eb3b
I20260806 16:14:45.693027 32754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap starting.
I20260806 16:14:45.695349 32754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:45.696106 32754 log.cc:824] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:14:45 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:45.698453 32754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: No bootstrap required, opened a new log
I20260806 16:14:45.702124 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:45.689429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.18.90.62:38827)
0806 16:14:45.689849 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:45.689865 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:45.689902 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:14:45.690526 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:45.690542 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:45.690862 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:45.691140 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:45.692059 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:45.693066 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:45.693934 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:45.694397 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:45.694567 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:45.697206 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:45.697212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:45.697229 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:45.697256 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:45.699114 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:45.699521 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:45.699527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:45.699533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:45.699684 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:45.699941 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:45.699944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:45.699946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:45.700495 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:45.700721 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:45.700935 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 16:14:45.701222 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20260806 16:14:45.702994 32754 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:45.703255 32754 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:45.703327 32754 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4763667d0c346918de89d738fa682dd, State: Initialized, Role: FOLLOWER
I20260806 16:14:45.703722 32754 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:45.703881 32754 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:14:45.703967 32754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:14:45.704062 32754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:45.705071 32754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:45.705404 32754 leader_election.cc:302] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4763667d0c346918de89d738fa682dd; no voters:
I20260806 16:14:45.705724 32754 leader_election.cc:288] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:14:45.705833 32759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:45.706074 32759 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 LEADER]: Becoming Leader. State: Replica: d4763667d0c346918de89d738fa682dd, State: Running, Role: LEADER
I20260806 16:14:45.706437 32759 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:45.706913 32754 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:45.708151 32761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4763667d0c346918de89d738fa682dd. Latest consensus state: current_term: 1 leader_uuid: "d4763667d0c346918de89d738fa682dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } } }
I20260806 16:14:45.708165 32760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4763667d0c346918de89d738fa682dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } } }
I20260806 16:14:45.708304 32761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:45.708315 32760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:45.708644 300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:45.711486 300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:45.716876 300 catalog_manager.cc:1347] Generated new cluster ID: 7620d98218274357b15abe30c96aa7f7
I20260806 16:14:45.716940 300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:14:45.726483 300 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:14:45.727689 300 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:14:45.733062 300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Generated new TSK 0
I20260806 16:14:45.733942 300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:14:45.764897 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38827
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:45.879375 312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:45.879668 312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:45.879720 312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:45.883999 312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:45.884063 312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:45.884120 312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:14:45.889369 312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:45.890381 312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:45.891428 312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:45.901084 317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:45.901264 312 server_base.cc:1034] running on GCE node
W20260806 16:14:45.901109 320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:45.901129 318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:45.901814 312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:45.902469 312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:45.903638 312 hybrid_clock.cc:648] HybridClock initialized: now 1786032885903606 us; error 42 us; skew 500 ppm
Aug 06 16:14:45 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:45.906483 312 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:14:45.907547 312 webserver.cc:460] Webserver started at http://127.18.90.1:44823/ using document root <none> and password file <none>
I20260806 16:14:45.908123 312 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:45.908221 312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:45.908517 312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:45.910605 312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "f039ad6842158c3d18deb52a0bea6c4c"
server_key_iv: "f5bd5be707013b9a944fbfd9d7d68154"
server_key_version: "encryptionkey@0"
I20260806 16:14:45.911193 312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "f039ad6842158c3d18deb52a0bea6c4c"
server_key_iv: "f5bd5be707013b9a944fbfd9d7d68154"
server_key_version: "encryptionkey@0"
I20260806 16:14:45.914983 312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:14:45.917465 327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:45.918697 312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:14:45.918841 312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "f039ad6842158c3d18deb52a0bea6c4c"
server_key_iv: "f5bd5be707013b9a944fbfd9d7d68154"
server_key_version: "encryptionkey@0"
I20260806 16:14:45.918952 312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:45.933391 312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:45.936395 312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:45.936604 312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:45.937230 312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:45.938078 312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:45.938125 312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:45.938215 312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:45.938246 312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:45.947139 312 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:34443
I20260806 16:14:45.947160 441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:34443 every 8 connection(s)
I20260806 16:14:45.948091 312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:14:45.952086 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 312
I20260806 16:14:45.952246 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:14:45.952579 18792 external_mini_cluster.cc:1428] Setting key da138742683fa61732f49f0021c04666
Aug 06 16:14:45 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:14:45.963970 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:45.950017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:58201 (local address 127.18.90.62:38827)
0806 16:14:45.950178 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:45.950183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:45.951144 (+ 961us) server_negotiation.cc:407] Connection header received
0806 16:14:45.952808 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:45.952812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:45.952871 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:45.952977 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:45.954393 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:45.955201 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:45.956492 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:45.956753 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:45.956870 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:45.959942 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:45.959945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:45.959947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:45.959974 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:45.961493 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:45.962181 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:45.962184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:45.962185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:45.962263 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:45.962669 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:45.962671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:45.962672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:45.962851 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:45.962927 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:45.963648 (+ 721us) server_negotiation.cc:299] Negotiation successful
0806 16:14:45.963766 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:14:45.965111 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:45.950484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:58201)
0806 16:14:45.950964 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:45.951022 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:45.952548 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:45.953231 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:45.953247 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:45.953622 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:45.954235 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:45.954258 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:45.955343 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:45.955345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:45.956369 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:45.956379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:45.956916 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:45.956920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:45.957096 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:45.957767 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:45.957785 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:45.959724 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:45.961623 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:45.961630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:45.961650 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:45.962063 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:45.962358 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:45.962365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:45.962367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:45.962562 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:45.962982 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:45.962989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:45.963372 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 16:14:45.963603 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 16:14:45.964008 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":141,"threads_started":1}
I20260806 16:14:45.966586 442 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:14:45.966845 442 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:45.967468 442 heartbeater.cc:511] Master 127.18.90.62:38827 requested a full tablet report, sending...
I20260806 16:14:45.969274 32718 ts_manager.cc:194] Registered new tserver with Master: e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1:34443)
I20260806 16:14:45.970708 32718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:58201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:14:45.983987 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38827
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:46.096709 447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:46.097066 447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:46.097147 447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:46.101511 447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:46.101578 447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:46.101639 447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:14:46.107053 447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:46.108229 447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:46.109405 447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:46.119065 455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:46.119240 452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:46.119073 453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:46.119273 447 server_base.cc:1034] running on GCE node
I20260806 16:14:46.119908 447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:46.120781 447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:46.121979 447 hybrid_clock.cc:648] HybridClock initialized: now 1786032886121964 us; error 26 us; skew 500 ppm
Aug 06 16:14:46 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032886, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:46.124922 447 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:14:46.125792 447 webserver.cc:460] Webserver started at http://127.18.90.2:42945/ using document root <none> and password file <none>
I20260806 16:14:46.126377 447 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:46.126470 447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:46.126706 447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:46.128813 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
format_stamp: "Formatted at 2026-08-06 16:14:46 on dist-test-slave-0b58"
server_key: "ae860c04e54c43c6635daea7508b44df"
server_key_iv: "26735b37814bcaa372e5881b42986fd3"
server_key_version: "encryptionkey@0"
I20260806 16:14:46.129369 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
format_stamp: "Formatted at 2026-08-06 16:14:46 on dist-test-slave-0b58"
server_key: "ae860c04e54c43c6635daea7508b44df"
server_key_iv: "26735b37814bcaa372e5881b42986fd3"
server_key_version: "encryptionkey@0"
I20260806 16:14:46.133203 447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:14:46.135542 462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:46.136760 447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:14:46.136893 447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
format_stamp: "Formatted at 2026-08-06 16:14:46 on dist-test-slave-0b58"
server_key: "ae860c04e54c43c6635daea7508b44df"
server_key_iv: "26735b37814bcaa372e5881b42986fd3"
server_key_version: "encryptionkey@0"
I20260806 16:14:46.137002 447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:46.973836 442 heartbeater.cc:503] Master 127.18.90.62:38827 was elected leader, sending a full tablet report...
W20260806 16:14:47.041209 447 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-0b58': real 0.885s user 0.004s sys 0.000s
I20260806 16:14:47.044132 447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:47.048818 447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:47.049158 447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:47.049844 447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:47.050954 447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:47.051023 447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:47.051093 447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:47.051127 447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:47.061473 447 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:36403
I20260806 16:14:47.061502 576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:36403 every 8 connection(s)
I20260806 16:14:47.062424 447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:14:47.065438 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 447
I20260806 16:14:47.065572 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:14:47.065881 18792 external_mini_cluster.cc:1428] Setting key 84ac262ecf6669ec4977848d7aa16ef5
Aug 06 16:14:47 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032886, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:14:47.079483 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.064536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:51687 (local address 127.18.90.62:38827)
0806 16:14:47.064801 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.064804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.067411 (+ 2607us) server_negotiation.cc:407] Connection header received
0806 16:14:47.067462 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.067464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.067516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.067610 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:47.068916 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.069448 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.071456 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.071687 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.071807 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.075265 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:47.075272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:47.075278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:47.075313 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:47.077217 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:47.077830 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:47.077833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:47.077834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:47.077910 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:47.078364 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:47.078367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:47.078368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:47.078543 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:47.078620 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.079238 (+ 618us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.079341 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260806 16:14:47.080500 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.065108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:51687)
0806 16:14:47.065729 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.065781 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.067153 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.067798 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.067809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.068179 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:47.068771 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.068787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.069591 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.069599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.071311 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.071323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.071845 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.071849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.072050 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.072792 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:47.072815 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:47.075033 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:47.077346 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:47.077355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:47.077374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:47.077728 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:47.078008 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:47.078013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:47.078017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:47.078233 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:47.078652 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:47.078660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:47.078999 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.079173 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.079440 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:14:47.081830 577 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38827
I20260806 16:14:47.082085 577 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:47.082571 577 heartbeater.cc:511] Master 127.18.90.62:38827 requested a full tablet report, sending...
I20260806 16:14:47.083719 32718 ts_manager.cc:194] Registered new tserver with Master: 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
I20260806 16:14:47.084585 32718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:51687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:14:47.095160 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38827
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:47.206298 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:47.206549 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:47.206640 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:47.210845 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:47.210906 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:47.210964 583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:14:47.216007 583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:47.217062 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:47.218047 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:47.227929 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:47.228016 583 server_base.cc:1034] running on GCE node
W20260806 16:14:47.227948 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:47.227948 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:47.228673 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:47.229307 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:47.230494 583 hybrid_clock.cc:648] HybridClock initialized: now 1786032887230485 us; error 27 us; skew 500 ppm
Aug 06 16:14:47 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032887, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:47.233275 583 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:14:47.234140 583 webserver.cc:460] Webserver started at http://127.18.90.3:46183/ using document root <none> and password file <none>
I20260806 16:14:47.234688 583 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:47.234781 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:47.235013 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:14:47.237090 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "1e8fa179d620489ebfe0ba8017563cb8"
format_stamp: "Formatted at 2026-08-06 16:14:47 on dist-test-slave-0b58"
server_key: "f270ea1b65ef687f2709b812e1e852ba"
server_key_iv: "d42c72b1f3e8fe6ad9aa68b5e6b25c4e"
server_key_version: "encryptionkey@0"
I20260806 16:14:47.237679 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "1e8fa179d620489ebfe0ba8017563cb8"
format_stamp: "Formatted at 2026-08-06 16:14:47 on dist-test-slave-0b58"
server_key: "f270ea1b65ef687f2709b812e1e852ba"
server_key_iv: "d42c72b1f3e8fe6ad9aa68b5e6b25c4e"
server_key_version: "encryptionkey@0"
I20260806 16:14:47.241371 583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:14:47.243811 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:47.245060 583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:14:47.245194 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "1e8fa179d620489ebfe0ba8017563cb8"
format_stamp: "Formatted at 2026-08-06 16:14:47 on dist-test-slave-0b58"
server_key: "f270ea1b65ef687f2709b812e1e852ba"
server_key_iv: "d42c72b1f3e8fe6ad9aa68b5e6b25c4e"
server_key_version: "encryptionkey@0"
I20260806 16:14:47.245312 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:47.270583 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:47.274021 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:47.274240 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:47.274856 583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:47.275696 583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:47.275769 583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:47.275846 583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:47.275899 583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:47.284996 583 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:33275
I20260806 16:14:47.285032 712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:33275 every 8 connection(s)
I20260806 16:14:47.285916 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:14:47.291393 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 583
I20260806 16:14:47.291525 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:14:47.291843 18792 external_mini_cluster.cc:1428] Setting key d85ac0314fc542550d239238cbc27890
Aug 06 16:14:47 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032887, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:47.300037 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.287758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:40097 (local address 127.18.90.62:38827)
0806 16:14:47.287938 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.287943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.288815 (+ 872us) server_negotiation.cc:407] Connection header received
0806 16:14:47.290177 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.290180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.290227 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.290302 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:47.291716 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.292384 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.293902 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.294197 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.294282 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.296713 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:47.296716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:47.296718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:47.296745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:47.298219 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:47.298664 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:47.298666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:47.298667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:47.298740 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:47.299028 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:47.299030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:47.299031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:47.299193 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:47.299265 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.299815 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.299919 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 16:14:47.300907 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.288141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:40097)
0806 16:14:47.288648 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.288695 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.289974 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.290471 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.290477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.290826 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:47.291505 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.291527 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.292496 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.292499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.293747 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.293758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.294337 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.294340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.294483 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.295024 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:47.295040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:47.296560 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:47.298317 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:47.298322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:47.298332 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:47.298580 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:47.298819 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:47.298821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:47.298822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:47.298951 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:47.299284 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:47.299290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:47.299573 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.299770 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.300028 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":174,"threads_started":1}
I20260806 16:14:47.302023 713 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38827
I20260806 16:14:47.302268 713 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:47.302745 713 heartbeater.cc:511] Master 127.18.90.62:38827 requested a full tablet report, sending...
I20260806 16:14:47.303758 32718 ts_manager.cc:194] Registered new tserver with Master: 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275)
I20260806 16:14:47.304383 32718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:40097
I20260806 16:14:47.305754 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:14:47.314718 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.307744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.18.90.62:38827)
0806 16:14:47.307882 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.307889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.308147 (+ 258us) server_negotiation.cc:407] Connection header received
0806 16:14:47.308243 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.308244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.308325 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.308389 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:47.309090 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.309569 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.310362 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.310566 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.310636 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.311910 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:47.311913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:47.311914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:47.311931 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:47.313146 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:47.313566 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:47.313568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:47.313569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:47.313635 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:47.313892 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:47.313895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:47.313896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:47.314106 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:47.314254 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.314431 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.314520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:14:47.317682 32718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:14:47.320667 32718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:14:47.334913 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.328705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36403 (local address 127.0.0.1:49652)
0806 16:14:47.329227 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.329274 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.329412 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.330585 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.330588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.330601 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:47.330677 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.330683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.332025 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.332028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.333425 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.333436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.334504 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.334508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.334608 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.334633 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.334680 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.334752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":132,"thread_start_us":101,"threads_started":1}
I20260806 16:14:47.334916 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.329709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:33275 (local address 127.0.0.1:34404)
0806 16:14:47.330392 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.330405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.330490 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.331222 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.331224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.331243 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:47.331318 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.331324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.332541 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.332545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.333585 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.333595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.334373 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.334376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.334477 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.334511 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.334618 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.334734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":30,"thread_start_us":118,"threads_started":1}
I20260806 16:14:47.335229 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.329482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.18.90.1:34443)
0806 16:14:47.329728 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.329735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.329767 (+ 32us) server_negotiation.cc:407] Connection header received
0806 16:14:47.329870 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.329876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.330100 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.330275 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:47.330805 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.331732 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.333745 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.334516 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.334609 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.334745 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.334985 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.335086 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260806 16:14:47.335343 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.328888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34443 (local address 127.0.0.1:34078)
0806 16:14:47.329397 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.329409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.329484 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.330483 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.330489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.330510 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:47.330652 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.330663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.331855 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.331858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.333425 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.333436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.334614 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.334617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.334706 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.334725 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.334816 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.334873 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":83,"thread_start_us":88,"threads_started":1}
I20260806 16:14:47.335582 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.328873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.18.90.2:36403)
0806 16:14:47.329992 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.330005 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.330028 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:14:47.330105 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.330110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.330448 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.330607 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:47.330763 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.331899 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.333555 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.334359 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.335136 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.335281 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.335335 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.335412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":122,"threads_started":1}
I20260806 16:14:47.335892 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.329921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.18.90.3:33275)
0806 16:14:47.330774 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.330786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.330820 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:14:47.330889 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.330894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.331069 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.331216 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:47.331441 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.332374 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.333711 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.334241 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.334363 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.334509 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.334846 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.334950 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":164,"threads_started":1}
I20260806 16:14:47.338035 376 tablet_service.cc:1467] Processing CreateTablet for tablet 2baba50baf134b66916cd27b404ed05a (DEFAULT_TABLE table=test-table [id=58006496de06438f8b776a8158f94944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:47.338035 511 tablet_service.cc:1467] Processing CreateTablet for tablet 2baba50baf134b66916cd27b404ed05a (DEFAULT_TABLE table=test-table [id=58006496de06438f8b776a8158f94944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:47.338954 511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baba50baf134b66916cd27b404ed05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:47.338987 376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baba50baf134b66916cd27b404ed05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:47.341085 647 tablet_service.cc:1467] Processing CreateTablet for tablet 2baba50baf134b66916cd27b404ed05a (DEFAULT_TABLE table=test-table [id=58006496de06438f8b776a8158f94944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:47.342099 647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baba50baf134b66916cd27b404ed05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:47.345477 731 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Bootstrap starting.
I20260806 16:14:47.347225 732 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap starting.
I20260806 16:14:47.347651 731 tablet_bootstrap.cc:654] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:47.348984 733 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Bootstrap starting.
I20260806 16:14:47.349125 731 log.cc:824] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:47.349496 732 tablet_bootstrap.cc:654] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:47.350433 732 log.cc:824] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:47.351387 731 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: No bootstrap required, opened a new log
I20260806 16:14:47.351591 731 ts_tablet_manager.cc:1397] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:14:47.351958 733 tablet_bootstrap.cc:654] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:47.352201 732 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: No bootstrap required, opened a new log
I20260806 16:14:47.352413 732 ts_tablet_manager.cc:1397] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:14:47.352768 733 log.cc:824] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:47.354678 733 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: No bootstrap required, opened a new log
I20260806 16:14:47.354916 733 ts_tablet_manager.cc:1397] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:14:47.355496 731 raft_consensus.cc:348] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.355784 731 raft_consensus.cc:374] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:47.355850 731 raft_consensus.cc:729] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c2469d28a84eb39e1573f6cf8d8d4a, State: Initialized, Role: FOLLOWER
I20260806 16:14:47.355822 732 raft_consensus.cc:348] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.356328 732 raft_consensus.cc:374] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:47.356297 731 consensus_queue.cc:260] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.356402 732 raft_consensus.cc:729] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8caa456eb0c4dbb971b02ba5ef8c31b, State: Initialized, Role: FOLLOWER
I20260806 16:14:47.356958 577 heartbeater.cc:503] Master 127.18.90.62:38827 was elected leader, sending a full tablet report...
I20260806 16:14:47.356918 732 consensus_queue.cc:260] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.357286 731 ts_tablet_manager.cc:1428] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:14:47.357856 732 ts_tablet_manager.cc:1428] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:14:47.359156 733 raft_consensus.cc:348] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.359409 733 raft_consensus.cc:374] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:47.359467 733 raft_consensus.cc:729] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8fa179d620489ebfe0ba8017563cb8, State: Initialized, Role: FOLLOWER
I20260806 16:14:47.359938 733 consensus_queue.cc:260] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.360827 713 heartbeater.cc:503] Master 127.18.90.62:38827 was elected leader, sending a full tablet report...
I20260806 16:14:47.360917 733 ts_tablet_manager.cc:1428] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 16:14:47.450083 443 tablet.cc:2369] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:14:47.536979 714 tablet.cc:2369] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:14:47.563660 578 tablet.cc:2369] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:14:47.680056 739 raft_consensus.cc:482] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:47.680382 739 raft_consensus.cc:504] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.681648 739 leader_election.cc:288] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403), 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275)
I20260806 16:14:47.686391 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.682042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:50831 (local address 127.18.90.3:33275)
0806 16:14:47.682195 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.682201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.682444 (+ 243us) server_negotiation.cc:407] Connection header received
0806 16:14:47.682910 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.682913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.682967 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.683038 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:47.683301 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.683827 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.685155 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.685768 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.685881 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.685950 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.686184 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.686271 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:14:47.686573 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.681969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:53341 (local address 127.18.90.2:36403)
0806 16:14:47.682122 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.682127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.682415 (+ 288us) server_negotiation.cc:407] Connection header received
0806 16:14:47.682614 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.682618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.682678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.682765 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:47.683049 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.683849 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.685381 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.685987 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.686122 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.686195 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.686393 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.686462 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:14:47.686595 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.681898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36403 (local address 127.18.90.1:53341)
0806 16:14:47.682303 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.682315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.682473 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.682780 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.682783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.682799 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:47.682926 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.682932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.683975 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.683978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.685249 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.685259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.686127 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.686145 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.686261 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.686285 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.686375 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.686443 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"spinlock_wait_cycles":26624,"thread_start_us":144,"threads_started":1}
I20260806 16:14:47.686971 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.681901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:33275 (local address 127.18.90.1:50831)
0806 16:14:47.682310 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.682323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:47.682480 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:47.683057 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:47.683060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:47.683073 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:47.683188 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.683194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.683938 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.683944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.685023 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:47.685033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.685902 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.685906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:47.686005 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.686032 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:14:47.686784 (+ 752us) client_negotiation.cc:240] Negotiation successful
0806 16:14:47.686857 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":96,"spinlock_wait_cycles":1408,"thread_start_us":116,"threads_started":1}
I20260806 16:14:47.687273 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baba50baf134b66916cd27b404ed05a" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" is_pre_election: true
I20260806 16:14:47.687582 531 raft_consensus.cc:2454] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 0.
I20260806 16:14:47.687868 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baba50baf134b66916cd27b404ed05a" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8fa179d620489ebfe0ba8017563cb8" is_pre_election: true
I20260806 16:14:47.688093 329 leader_election.cc:302] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c2469d28a84eb39e1573f6cf8d8d4a, e8caa456eb0c4dbb971b02ba5ef8c31b; no voters:
I20260806 16:14:47.688227 667 raft_consensus.cc:2454] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 0.
I20260806 16:14:47.688582 739 raft_consensus.cc:2781] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:14:47.688671 739 raft_consensus.cc:482] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:14:47.688712 739 raft_consensus.cc:3037] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:47.690022 739 raft_consensus.cc:504] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.690398 739 leader_election.cc:288] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 election: Requested vote from peers 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403), 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275)
I20260806 16:14:47.690743 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baba50baf134b66916cd27b404ed05a" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
I20260806 16:14:47.690891 531 raft_consensus.cc:3037] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:47.690950 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baba50baf134b66916cd27b404ed05a" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8fa179d620489ebfe0ba8017563cb8"
I20260806 16:14:47.691071 667 raft_consensus.cc:3037] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:47.691998 531 raft_consensus.cc:2454] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 1.
I20260806 16:14:47.692286 667 raft_consensus.cc:2454] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 1.
I20260806 16:14:47.692318 329 leader_election.cc:302] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c2469d28a84eb39e1573f6cf8d8d4a, e8caa456eb0c4dbb971b02ba5ef8c31b; no voters:
I20260806 16:14:47.692576 739 raft_consensus.cc:2781] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:47.692799 739 raft_consensus.cc:686] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 LEADER]: Becoming Leader. State: Replica: e8caa456eb0c4dbb971b02ba5ef8c31b, State: Running, Role: LEADER
I20260806 16:14:47.693138 739 consensus_queue.cc:237] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:47.696537 32718 catalog_manager.cc:5495] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1). New cstate: current_term: 1 leader_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } health_report { overall_health: UNKNOWN } } }
I20260806 16:14:47.786855 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.783530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.18.90.1:34443)
0806 16:14:47.783660 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:47.783663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:47.783679 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:14:47.783795 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:47.783797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:47.783857 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:47.783985 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:14:47.784203 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.784704 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.785487 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:47.785696 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:47.785771 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:47.786021 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:14:47.786445 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:14:47.786533 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:47.786590 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 16:14:47.786639 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:14:47.792917 531 raft_consensus.cc:1261] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Refusing update from remote peer e8caa456eb0c4dbb971b02ba5ef8c31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:47.792917 667 raft_consensus.cc:1261] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Refusing update from remote peer e8caa456eb0c4dbb971b02ba5ef8c31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:47.793695 743 consensus_queue.cc:1035] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:47.793875 739 consensus_queue.cc:1035] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:47.802939 748 mvcc.cc:205] Tried to move back new op lower bound from 7315590708392173568 to 7315590707993800704. Current Snapshot: MvccSnapshot[applied={T|T < 7315590708392173568}]
I20260806 16:14:47.811193 32717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:14:47.811398 32717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:14:47.813851 32717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Sending DeleteTablet for 3 replicas of tablet 2baba50baf134b66916cd27b404ed05a
I20260806 16:14:47.814695 511 tablet_service.cc:1514] Processing DeleteTablet for tablet 2baba50baf134b66916cd27b404ed05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:47 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:49652
I20260806 16:14:47.815210 755 tablet_replica.cc:330] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: stopping tablet replica
I20260806 16:14:47.815227 647 tablet_service.cc:1514] Processing DeleteTablet for tablet 2baba50baf134b66916cd27b404ed05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:47 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34404
I20260806 16:14:47.815572 755 raft_consensus.cc:2229] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:14:47.815763 756 tablet_replica.cc:330] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: stopping tablet replica
I20260806 16:14:47.815904 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 312
I20260806 16:14:47.815903 755 raft_consensus.cc:2258] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:47.816183 756 raft_consensus.cc:2229] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:14:47.816567 756 raft_consensus.cc:2258] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:47.818089 756 ts_tablet_manager.cc:1905] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:14:47.818813 755 ts_tablet_manager.cc:1905] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:14:47.821204 756 ts_tablet_manager.cc:1918] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:14:47.821327 756 log.cc:1196] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/2baba50baf134b66916cd27b404ed05a
I20260806 16:14:47.821585 755 ts_tablet_manager.cc:1918] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:14:47.821669 756 ts_tablet_manager.cc:1939] T 2baba50baf134b66916cd27b404ed05a P 1e8fa179d620489ebfe0ba8017563cb8: Deleting consensus metadata
I20260806 16:14:47.821691 755 log.cc:1196] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/2baba50baf134b66916cd27b404ed05a
I20260806 16:14:47.822003 755 ts_tablet_manager.cc:1939] T 2baba50baf134b66916cd27b404ed05a P 51c2469d28a84eb39e1573f6cf8d8d4a: Deleting consensus metadata
I20260806 16:14:47.823101 32703 catalog_manager.cc:4841] TS 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275): tablet 2baba50baf134b66916cd27b404ed05a (table test-table [id=58006496de06438f8b776a8158f94944]) successfully deleted
I20260806 16:14:47.823354 32702 catalog_manager.cc:4841] TS 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403): tablet 2baba50baf134b66916cd27b404ed05a (table test-table [id=58006496de06438f8b776a8158f94944]) successfully deleted
W20260806 16:14:47.823822 32704 connection.cc:535] client connection to 127.18.90.1:34443 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 16:14:47.823969 32704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:14:47.824791 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 447
I20260806 16:14:47.824988 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:47.824657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34443 (local address 127.0.0.1:34104)
0806 16:14:47.824799 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:47.824882 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 16:14:47.826642 32704 catalog_manager.cc:4584] TS e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1:34443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2baba50baf134b66916cd27b404ed05a: Network error: Client connection negotiation failed: client connection to 127.18.90.1:34443: connect: Connection refused (error 111)
I20260806 16:14:47.831925 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 583
I20260806 16:14:47.837832 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 32685
I20260806 16:14:47.844856 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38827
--webserver_interface=127.18.90.62
--webserver_port=44499
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:47.954790 758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:47.955084 758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:47.955156 758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:47.958988 758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:47.959075 758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:47.959108 758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:47.959141 758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:47.959192 758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:47.964298 758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:38827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=44499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:47.965365 758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:47.966481 758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:47.982029 767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:47.982028 765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:47.982049 764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:47.982072 758 server_base.cc:1034] running on GCE node
I20260806 16:14:47.982933 758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:47.984094 758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:47.985253 758 hybrid_clock.cc:648] HybridClock initialized: now 1786032887985233 us; error 35 us; skew 500 ppm
Aug 06 16:14:47 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032887, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:47.987998 758 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:47.988919 758 webserver.cc:460] Webserver started at http://127.18.90.62:44499/ using document root <none> and password file <none>
I20260806 16:14:47.989475 758 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:47.989566 758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:47.993309 758 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 16:14:47.995270 774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:47.996423 758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:14:47.996558 758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "d4763667d0c346918de89d738fa682dd"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "ca06b48a362b76b0b4933e843482c111"
server_key_iv: "c249f4345ff80707ab98f3e1c723cf9e"
server_key_version: "encryptionkey@0"
I20260806 16:14:47.997107 758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:48.007133 758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:48.009850 758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:48.010047 758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:48.017213 758 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:38827
I20260806 16:14:48.017549 827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:38827 every 8 connection(s)
I20260806 16:14:48.018354 758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:48.021086 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 758
I20260806 16:14:48.021572 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:34443
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:38827
--webserver_port=44823
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:48.023288 828 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:14:48.025089 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap starting.
I20260806 16:14:48.042033 828 log.cc:824] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:48.050575 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:14:48.050958 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap complete.
I20260806 16:14:48.054993 828 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:48.055374 828 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4763667d0c346918de89d738fa682dd, State: Initialized, Role: FOLLOWER
I20260806 16:14:48.055902 828 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:48.056016 828 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:14:48.056109 828 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:14:48.056216 828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:14:48.058130 828 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:48.058473 828 leader_election.cc:302] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4763667d0c346918de89d738fa682dd; no voters:
I20260806 16:14:48.058794 828 leader_election.cc:288] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:14:48.058888 832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:14:48.059067 832 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 LEADER]: Becoming Leader. State: Replica: d4763667d0c346918de89d738fa682dd, State: Running, Role: LEADER
I20260806 16:14:48.059403 832 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:48.060025 828 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:48.060457 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4763667d0c346918de89d738fa682dd. Latest consensus state: current_term: 2 leader_uuid: "d4763667d0c346918de89d738fa682dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } } }
I20260806 16:14:48.060649 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:48.060833 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d4763667d0c346918de89d738fa682dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } } }
I20260806 16:14:48.060948 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:48.061375 837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:48.066042 837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:48.066913 837 catalog_manager.cc:1259] Loaded cluster ID: 7620d98218274357b15abe30c96aa7f7
I20260806 16:14:48.066979 837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:14:48.068403 837 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 16:14:48.073223 837 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 16:14:48.078896 837 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:14:48.079666 837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Loaded TSK: 0
I20260806 16:14:48.080927 837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:14:48.135619 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:48.135918 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:48.135998 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:48.139955 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:48.140048 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:48.140138 830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:14:48.145653 830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:34443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=44823
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:48.146773 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:48.147831 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:48.158665 855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:48.158674 858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:48.159305 856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:48.159495 830 server_base.cc:1034] running on GCE node
I20260806 16:14:48.159974 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:48.160678 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:48.161880 830 hybrid_clock.cc:648] HybridClock initialized: now 1786032888161839 us; error 52 us; skew 500 ppm
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:48.164665 830 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:14:48.165553 830 webserver.cc:460] Webserver started at http://127.18.90.1:44823/ using document root <none> and password file <none>
I20260806 16:14:48.166105 830 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:48.166191 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:48.170254 830 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:14:48.172554 866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.173854 830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:48.174002 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "f039ad6842158c3d18deb52a0bea6c4c"
server_key_iv: "f5bd5be707013b9a944fbfd9d7d68154"
server_key_version: "encryptionkey@0"
I20260806 16:14:48.174566 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:48.190162 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:48.193322 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:48.193535 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:48.194106 830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:48.195204 873 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:14:48.200567 830 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:14:48.200672 830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 16:14:48.200836 830 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:14:48.202487 830 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:14:48.202557 830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 16:14:48.202814 873 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap starting.
I20260806 16:14:48.213397 830 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:34443
I20260806 16:14:48.214376 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:14:48.214943 980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:34443 every 8 connection(s)
I20260806 16:14:48.219058 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 830
I20260806 16:14:48.219560 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:36403
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:38827
--webserver_port=42945
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:48.222442 873 log.cc:824] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:48.233058 873 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:14:48.233528 873 tablet_bootstrap.cc:492] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap complete.
I20260806 16:14:48.234997 873 ts_tablet_manager.cc:1397] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.016s
I20260806 16:14:48.235285 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.216499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:36701)
0806 16:14:48.217077 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.217129 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.218049 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.219319 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.219328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.219807 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:48.220615 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.220637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.222696 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.222702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.224181 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.224191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.225482 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.225486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.225669 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.226418 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:48.226441 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:48.228829 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:48.231640 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.231645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.231657 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.231988 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.232459 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.232463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.232464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.232637 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.233420 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:48.233429 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:48.233722 (+ 293us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.233913 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.234265 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":133,"threads_started":1}
I20260806 16:14:48.235325 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.216533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:36701 (local address 127.18.90.62:38827)
0806 16:14:48.217627 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.217645 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.217691 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:14:48.218493 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.218504 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.218847 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.219213 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:48.220829 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.222496 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.224396 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.225097 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.225273 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.229086 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:48.229094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:48.229115 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:48.229157 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:48.231494 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.232136 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.232146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.232153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.232347 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.232765 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.232770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.232773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.233289 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:48.233440 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.234091 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.234433 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":119,"threads_started":1}
I20260806 16:14:48.237794 873 raft_consensus.cc:348] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:48.238308 873 raft_consensus.cc:729] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8caa456eb0c4dbb971b02ba5ef8c31b, State: Initialized, Role: FOLLOWER
I20260806 16:14:48.238445 981 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38827
I20260806 16:14:48.238686 981 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:48.238862 873 consensus_queue.cc:260] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } } peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } }
I20260806 16:14:48.239220 981 heartbeater.cc:511] Master 127.18.90.62:38827 requested a full tablet report, sending...
I20260806 16:14:48.239874 873 ts_tablet_manager.cc:1428] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:14:48.241926 791 ts_manager.cc:194] Registered new tserver with Master: e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1:34443)
I20260806 16:14:48.245801 791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:36701
I20260806 16:14:48.248155 981 heartbeater.cc:503] Master 127.18.90.62:38827 was elected leader, sending a full tablet report...
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032887, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:14:48.255681 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.245212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.18.90.1:34443)
0806 16:14:48.245970 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.245983 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.246004 (+ 21us) server_negotiation.cc:407] Connection header received
0806 16:14:48.246065 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.246069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.246229 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.246376 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:48.247163 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.248243 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.249357 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.249687 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.249812 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.252209 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:48.252217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:48.252226 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:48.252249 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:48.253831 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.254309 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.254315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.254317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.254398 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.254674 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.254678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.254680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.254914 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:48.255046 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.255388 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.255539 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":98,"threads_started":1}
I20260806 16:14:48.255682 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.245155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34443 (local address 127.0.0.1:34112)
0806 16:14:48.245553 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.245595 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.245712 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.246379 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.246385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.246900 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:48.247033 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.247042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.248424 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.248428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.249225 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.249234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.249825 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.249829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.249953 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.250550 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:48.250563 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:48.252064 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:48.253947 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.253958 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.253963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.254188 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.254492 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.254495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.254497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.254586 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.255040 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:48.255045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:48.255163 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.255308 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.255501 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":125,"threads_started":1}
I20260806 16:14:48.256754 913 tablet_service.cc:1514] Processing DeleteTablet for tablet 2baba50baf134b66916cd27b404ed05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:47 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34112
I20260806 16:14:48.257719 990 tablet_replica.cc:330] stopping tablet replica
I20260806 16:14:48.258011 990 raft_consensus.cc:2229] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:14:48.258260 990 raft_consensus.cc:2258] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:48.259177 990 ts_tablet_manager.cc:1905] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:14:48.263180 990 ts_tablet_manager.cc:1918] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:14:48.263305 990 log.cc:1196] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/2baba50baf134b66916cd27b404ed05a
I20260806 16:14:48.263607 990 ts_tablet_manager.cc:1939] T 2baba50baf134b66916cd27b404ed05a P e8caa456eb0c4dbb971b02ba5ef8c31b: Deleting consensus metadata
I20260806 16:14:48.265316 778 catalog_manager.cc:4841] TS e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1:34443): tablet 2baba50baf134b66916cd27b404ed05a (table test-table [id=58006496de06438f8b776a8158f94944]) successfully deleted
W20260806 16:14:48.342180 985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:48.342413 985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:48.342453 985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:48.346412 985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:48.346470 985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:48.346561 985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:14:48.351804 985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:36403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=42945
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:48.352914 985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:48.353955 985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:48.363786 998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:48.363946 985 server_base.cc:1034] running on GCE node
W20260806 16:14:48.363803 996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:48.364153 995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:48.364634 985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:48.365476 985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:48.372853 985 hybrid_clock.cc:648] HybridClock initialized: now 1786032888372834 us; error 58 us; skew 500 ppm
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:48.375779 985 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:14:48.376807 985 webserver.cc:460] Webserver started at http://127.18.90.2:42945/ using document root <none> and password file <none>
I20260806 16:14:48.377374 985 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:48.377485 985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:48.381460 985 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:14:48.383419 1005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.384660 985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:14:48.384781 985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
format_stamp: "Formatted at 2026-08-06 16:14:46 on dist-test-slave-0b58"
server_key: "ae860c04e54c43c6635daea7508b44df"
server_key_iv: "26735b37814bcaa372e5881b42986fd3"
server_key_version: "encryptionkey@0"
I20260806 16:14:48.385324 985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:48.401033 985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:48.403759 985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:48.403960 985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:48.404601 985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:48.405426 985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:48.405473 985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.405542 985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:48.405581 985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.414389 985 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:36403
I20260806 16:14:48.414434 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:36403 every 8 connection(s)
I20260806 16:14:48.415328 985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:14:48.419448 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 985
I20260806 16:14:48.419852 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:33275
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:38827
--webserver_port=46183
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:48.428922 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.417011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:37599 (local address 127.18.90.62:38827)
0806 16:14:48.417154 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.417157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.417895 (+ 738us) server_negotiation.cc:407] Connection header received
0806 16:14:48.419330 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.419333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.419398 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.419504 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:48.420803 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.421342 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.422490 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.422754 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.422867 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.425362 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:48.425366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:48.425367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:48.425392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:48.426836 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.427347 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.427352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.427353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.427450 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.427797 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.427800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.427802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.427972 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:48.428078 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.428608 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.428748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:14:48.429651 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.417319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:37599)
0806 16:14:48.417747 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.417787 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.419106 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.419679 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.419686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.420104 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:48.420650 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.420664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.421475 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.421484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.422358 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.422368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.422918 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.422923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.423086 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.423681 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:48.423699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:48.425174 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:48.426957 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.426962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.426975 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.427240 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.427544 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.427547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.427549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.427683 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.428078 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:48.428085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:48.428383 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.428568 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.428871 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260806 16:14:48.430847 1121 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38827
I20260806 16:14:48.431131 1121 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:48.431686 1121 heartbeater.cc:511] Master 127.18.90.62:38827 requested a full tablet report, sending...
I20260806 16:14:48.432799 791 ts_manager.cc:194] Registered new tserver with Master: 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
I20260806 16:14:48.433544 791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:37599
W20260806 16:14:48.533954 1124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:48.534196 1124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:48.534236 1124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:48.538230 1124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:48.538285 1124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:48.538340 1124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:14:48.543431 1124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:33275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=46183
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:48.544502 1124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:48.545480 1124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:48.554950 1132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:48.554946 1129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:48.555191 1130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:48.555197 1124 server_base.cc:1034] running on GCE node
I20260806 16:14:48.555663 1124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:48.556406 1124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:48.557603 1124 hybrid_clock.cc:648] HybridClock initialized: now 1786032888557580 us; error 42 us; skew 500 ppm
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:48.560531 1124 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:14:48.561528 1124 webserver.cc:460] Webserver started at http://127.18.90.3:46183/ using document root <none> and password file <none>
I20260806 16:14:48.562080 1124 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:48.562186 1124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:48.566044 1124 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 16:14:48.568154 1139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.569490 1124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:14:48.569640 1124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "1e8fa179d620489ebfe0ba8017563cb8"
format_stamp: "Formatted at 2026-08-06 16:14:47 on dist-test-slave-0b58"
server_key: "f270ea1b65ef687f2709b812e1e852ba"
server_key_iv: "d42c72b1f3e8fe6ad9aa68b5e6b25c4e"
server_key_version: "encryptionkey@0"
I20260806 16:14:48.570202 1124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:48.581307 1124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:48.584075 1124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:48.584335 1124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:48.584924 1124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:48.585736 1124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:48.585807 1124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.585878 1124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:48.585932 1124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:48.595284 1124 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:33275
I20260806 16:14:48.595311 1254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:33275 every 8 connection(s)
I20260806 16:14:48.596340 1124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:14:48.596737 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1124
I20260806 16:14:48.605845 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.597542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.18.90.62:38827)
0806 16:14:48.597698 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.597703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.597922 (+ 219us) server_negotiation.cc:407] Connection header received
0806 16:14:48.598050 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.598053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.598095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.598171 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:48.599068 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.599766 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.600699 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.601030 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.601132 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.602610 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:48.602618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:48.602620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:48.602642 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:48.604092 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.604610 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.604612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.604614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.604686 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.604962 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.604964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.604965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.605119 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:48.605267 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.605503 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.605617 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 16:14:48 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032888, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:14:48.612022 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.598463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:33497 (local address 127.18.90.62:38827)
0806 16:14:48.598681 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.598684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.599767 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 16:14:48.601549 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.601552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.601593 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.601699 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:48.602921 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.603731 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.605358 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.605674 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.605783 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.608755 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:48.608757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:48.608759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:48.608780 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:48.610206 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.610670 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.610672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.610673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.610750 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:48.611098 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:48.611100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:48.611101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:48.611260 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:48.611332 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.611805 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.611899 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260806 16:14:48.612944 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.599065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:33497)
0806 16:14:48.599565 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.599619 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.601312 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.601923 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.601931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.602295 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:48.602773 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.602787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.603863 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.603866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.605233 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.605244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.605985 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.605993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.606277 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.606863 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:48.606890 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:48.608597 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:48.610315 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.610320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.610332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.610575 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.610860 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:48.610862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:48.610864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:48.611004 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:48.611350 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:48.611355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:48.611579 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.611739 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.611994 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":106,"threads_started":1}
I20260806 16:14:48.614073 1255 heartbeater.cc:347] Connected to a master server at 127.18.90.62:38827
I20260806 16:14:48.614305 1255 heartbeater.cc:464] Registering TS with master...
I20260806 16:14:48.614743 1255 heartbeater.cc:511] Master 127.18.90.62:38827 requested a full tablet report, sending...
I20260806 16:14:48.615665 791 ts_manager.cc:194] Registered new tserver with Master: 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275)
I20260806 16:14:48.616214 791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:33497
I20260806 16:14:48.619557 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:14:48.623674 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.620150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.18.90.62:38827)
0806 16:14:48.620316 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.620321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.620534 (+ 213us) server_negotiation.cc:407] Connection header received
0806 16:14:48.620656 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.620659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.620746 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.620828 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:14:48.621040 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.621544 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.622365 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.622577 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.622654 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.622920 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:14:48.623316 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:14:48.623404 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.623521 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.623577 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:14:48.624322 791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:14:48.627630 791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:14:48.635777 913 tablet_service.cc:1467] Processing CreateTablet for tablet e77946b72b9e4db7a65db8bdcd8b77a7 (DEFAULT_TABLE table=test-table [id=76491abbf7a94c1cb0c0c6c36d026c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:48.636525 913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77946b72b9e4db7a65db8bdcd8b77a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:48.639178 873 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap starting.
I20260806 16:14:48.640105 873 tablet_bootstrap.cc:654] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:48.640887 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.634678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:33275 (local address 127.0.0.1:34408)
0806 16:14:48.634806 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.634823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.634897 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.636284 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.636287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.636319 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:48.636448 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.636458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.638156 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.638159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.639164 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.639176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.640420 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.640424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.640537 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.640579 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.640660 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.640742 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":47,"spinlock_wait_cycles":7552}
I20260806 16:14:48.640931 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.635112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.18.90.3:33275)
0806 16:14:48.635699 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.635712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.635739 (+ 27us) server_negotiation.cc:407] Connection header received
0806 16:14:48.635823 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.635829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.636080 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.636793 (+ 713us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:48.636942 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.638034 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.639379 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.640286 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.640420 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.640577 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.640689 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.640789 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":204,"threads_started":1}
I20260806 16:14:48.641096 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.635000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36403 (local address 127.0.0.1:49660)
0806 16:14:48.635250 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.635265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.635356 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.636284 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.636287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.636318 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:48.636626 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.636632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.638119 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.638123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.639347 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.639359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.640577 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.640580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.640712 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.640746 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.640814 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.640899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":241,"spinlock_wait_cycles":4864,"thread_start_us":120,"threads_started":1}
I20260806 16:14:48.641782 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.635432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.18.90.2:36403)
0806 16:14:48.635798 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.635818 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.635835 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:14:48.635899 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.635903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.636079 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.636405 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:48.636718 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.637962 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.639501 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.640409 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.641303 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.641473 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.641529 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.641621 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
I20260806 16:14:48.641814 873 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: No bootstrap required, opened a new log
I20260806 16:14:48.641893 873 ts_tablet_manager.cc:1397] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:14:48.642333 873 raft_consensus.cc:348] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.642453 873 raft_consensus.cc:374] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:48.642491 873 raft_consensus.cc:729] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8caa456eb0c4dbb971b02ba5ef8c31b, State: Initialized, Role: FOLLOWER
I20260806 16:14:48.643966 1188 tablet_service.cc:1467] Processing CreateTablet for tablet e77946b72b9e4db7a65db8bdcd8b77a7 (DEFAULT_TABLE table=test-table [id=76491abbf7a94c1cb0c0c6c36d026c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:48.644330 1055 tablet_service.cc:1467] Processing CreateTablet for tablet e77946b72b9e4db7a65db8bdcd8b77a7 (DEFAULT_TABLE table=test-table [id=76491abbf7a94c1cb0c0c6c36d026c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:14:48.645023 1188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77946b72b9e4db7a65db8bdcd8b77a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:48.645306 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77946b72b9e4db7a65db8bdcd8b77a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:14:48.651703 873 consensus_queue.cc:260] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.652037 873 ts_tablet_manager.cc:1428] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260806 16:14:48.652451 1268 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Bootstrap starting.
I20260806 16:14:48.652904 1269 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Bootstrap starting.
I20260806 16:14:48.654757 1268 tablet_bootstrap.cc:654] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:48.655490 1269 tablet_bootstrap.cc:654] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Neither blocks nor log segments found. Creating new log.
I20260806 16:14:48.655499 1268 log.cc:824] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:48.656155 1269 log.cc:824] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:48.657594 1268 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: No bootstrap required, opened a new log
I20260806 16:14:48.657840 1268 ts_tablet_manager.cc:1397] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:14:48.658059 1269 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: No bootstrap required, opened a new log
I20260806 16:14:48.658272 1269 ts_tablet_manager.cc:1397] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:14:48.661027 1268 raft_consensus.cc:348] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.661286 1268 raft_consensus.cc:374] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:48.661343 1268 raft_consensus.cc:729] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8fa179d620489ebfe0ba8017563cb8, State: Initialized, Role: FOLLOWER
I20260806 16:14:48.661185 1269 raft_consensus.cc:348] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.661465 1269 raft_consensus.cc:374] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:14:48.661527 1269 raft_consensus.cc:729] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c2469d28a84eb39e1573f6cf8d8d4a, State: Initialized, Role: FOLLOWER
I20260806 16:14:48.661813 1268 consensus_queue.cc:260] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.661927 1269 consensus_queue.cc:260] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.662484 1255 heartbeater.cc:503] Master 127.18.90.62:38827 was elected leader, sending a full tablet report...
I20260806 16:14:48.662580 1121 heartbeater.cc:503] Master 127.18.90.62:38827 was elected leader, sending a full tablet report...
I20260806 16:14:48.662914 1268 ts_tablet_manager.cc:1428] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260806 16:14:48.664106 1269 ts_tablet_manager.cc:1428] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260806 16:14:48.666215 1122 tablet.cc:2369] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:14:48.716032 982 tablet.cc:2369] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:14:48.847450 1256 tablet.cc:2369] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:14:48.854967 987 raft_consensus.cc:482] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:48.855178 987 raft_consensus.cc:504] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.856323 987 leader_election.cc:288] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275), 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
I20260806 16:14:48.860633 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.856780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:48377 (local address 127.18.90.2:36403)
0806 16:14:48.856954 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.856959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.857140 (+ 181us) server_negotiation.cc:407] Connection header received
0806 16:14:48.857284 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.857287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.857334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.857415 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:48.857680 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.858200 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.859513 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.860029 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.860145 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.860206 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.860433 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.860501 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:14:48.860661 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.856551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36403 (local address 127.18.90.1:48377)
0806 16:14:48.857052 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.857064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.857194 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.857426 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.857428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.857455 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:48.857577 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.857581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.858310 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.858316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.859385 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.859395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.860149 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.860152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.860245 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.860315 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.860414 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.860485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":5,"thread_start_us":85,"threads_started":1}
I20260806 16:14:48.860673 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.856717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:50937 (local address 127.18.90.3:33275)
0806 16:14:48.856860 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.856865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.857023 (+ 158us) server_negotiation.cc:407] Connection header received
0806 16:14:48.857242 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.857249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.857294 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.857379 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:14:48.857702 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.858406 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.859533 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.860077 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.860197 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.860256 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.860472 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.860542 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:14:48.860709 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.856563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:33275 (local address 127.18.90.1:50937)
0806 16:14:48.856926 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.856939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:48.857084 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:48.857392 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:48.857395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:48.857412 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:14:48.857579 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.857584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.858521 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.858524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.859403 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:48.859413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.860210 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.860214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:48.860345 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.860362 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:14:48.860453 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:14:48.860525 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":49,"spinlock_wait_cycles":19712,"thread_start_us":98,"threads_started":1}
I20260806 16:14:48.861349 1208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77946b72b9e4db7a65db8bdcd8b77a7" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8fa179d620489ebfe0ba8017563cb8" is_pre_election: true
I20260806 16:14:48.861469 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77946b72b9e4db7a65db8bdcd8b77a7" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" is_pre_election: true
I20260806 16:14:48.861647 1208 raft_consensus.cc:2454] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 0.
I20260806 16:14:48.861785 1075 raft_consensus.cc:2454] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 0.
I20260806 16:14:48.862141 869 leader_election.cc:302] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8fa179d620489ebfe0ba8017563cb8, e8caa456eb0c4dbb971b02ba5ef8c31b; no voters:
I20260806 16:14:48.862442 987 raft_consensus.cc:2781] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:14:48.862528 987 raft_consensus.cc:482] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:14:48.862571 987 raft_consensus.cc:3037] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:48.863580 987 raft_consensus.cc:504] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.863941 987 leader_election.cc:288] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 election: Requested vote from peers 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275), 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
I20260806 16:14:48.864347 1208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77946b72b9e4db7a65db8bdcd8b77a7" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8fa179d620489ebfe0ba8017563cb8"
I20260806 16:14:48.864360 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77946b72b9e4db7a65db8bdcd8b77a7" candidate_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
I20260806 16:14:48.864463 1208 raft_consensus.cc:3037] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:48.864492 1075 raft_consensus.cc:3037] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:14:48.865553 1075 raft_consensus.cc:2454] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 1.
I20260806 16:14:48.865553 1208 raft_consensus.cc:2454] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8caa456eb0c4dbb971b02ba5ef8c31b in term 1.
I20260806 16:14:48.865851 868 leader_election.cc:302] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c2469d28a84eb39e1573f6cf8d8d4a, e8caa456eb0c4dbb971b02ba5ef8c31b; no voters:
I20260806 16:14:48.866041 987 raft_consensus.cc:2781] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:14:48.866289 987 raft_consensus.cc:686] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 LEADER]: Becoming Leader. State: Replica: e8caa456eb0c4dbb971b02ba5ef8c31b, State: Running, Role: LEADER
I20260806 16:14:48.866568 987 consensus_queue.cc:237] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:48.869573 790 catalog_manager.cc:5495] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1). New cstate: current_term: 1 leader_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } health_report { overall_health: HEALTHY } } }
I20260806 16:14:48.934419 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.930913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.18.90.1:34443)
0806 16:14:48.931232 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:48.931240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:48.931253 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:14:48.931337 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:48.931339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:48.931392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:48.931494 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:14:48.931703 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.932209 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.933033 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:48.933249 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:48.933343 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:48.933582 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:14:48.934003 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:14:48.934084 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:48.934146 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 16:14:48.934193 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":148,"threads_started":1}
I20260806 16:14:48.938506 1208 raft_consensus.cc:1261] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Refusing update from remote peer e8caa456eb0c4dbb971b02ba5ef8c31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:48.938506 1075 raft_consensus.cc:1261] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Refusing update from remote peer e8caa456eb0c4dbb971b02ba5ef8c31b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:14:48.939262 1276 consensus_queue.cc:1035] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:48.939489 987 consensus_queue.cc:1035] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:14:48.947908 1281 mvcc.cc:205] Tried to move back new op lower bound from 7315590713084723200 to 7315590712799891456. Current Snapshot: MvccSnapshot[applied={T|T < 7315590713084723200}]
I20260806 16:14:48.954836 789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:14:48.955041 789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:14:48.957448 789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Sending DeleteTablet for 3 replicas of tablet e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:14:48.958163 1188 tablet_service.cc:1514] Processing DeleteTablet for tablet e77946b72b9e4db7a65db8bdcd8b77a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:48 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34408
I20260806 16:14:48.958211 1055 tablet_service.cc:1514] Processing DeleteTablet for tablet e77946b72b9e4db7a65db8bdcd8b77a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:48 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:49660
I20260806 16:14:48.958875 913 tablet_service.cc:1514] Processing DeleteTablet for tablet e77946b72b9e4db7a65db8bdcd8b77a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:14:48 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34112
I20260806 16:14:48.959048 1290 tablet_replica.cc:330] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: stopping tablet replica
I20260806 16:14:48.959226 1291 tablet_replica.cc:330] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: stopping tablet replica
I20260806 16:14:48.959349 1291 raft_consensus.cc:2229] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:14:48.959372 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 830
I20260806 16:14:48.959388 1290 raft_consensus.cc:2229] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:14:48.959698 1290 raft_consensus.cc:2258] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:48.959700 1291 raft_consensus.cc:2258] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:48.960561 1289 tablet_replica.cc:330] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: stopping tablet replica
I20260806 16:14:48.960826 1289 raft_consensus.cc:2229] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:14:48.961076 1289 raft_consensus.cc:2258] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:14:48.961804 1290 ts_tablet_manager.cc:1905] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:14:48.962524 1289 ts_tablet_manager.cc:1905] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:14:48.964150 1290 ts_tablet_manager.cc:1918] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:14:48.964273 1290 log.cc:1196] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:14:48.964576 1290 ts_tablet_manager.cc:1939] T e77946b72b9e4db7a65db8bdcd8b77a7 P 51c2469d28a84eb39e1573f6cf8d8d4a: Deleting consensus metadata
I20260806 16:14:48.965745 776 catalog_manager.cc:4841] TS 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403): tablet e77946b72b9e4db7a65db8bdcd8b77a7 (table test-table [id=76491abbf7a94c1cb0c0c6c36d026c45]) successfully deleted
I20260806 16:14:48.966092 1289 ts_tablet_manager.cc:1918] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:14:48.966259 1289 log.cc:1196] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:14:48.966635 1289 ts_tablet_manager.cc:1939] T e77946b72b9e4db7a65db8bdcd8b77a7 P 1e8fa179d620489ebfe0ba8017563cb8: Deleting consensus metadata
I20260806 16:14:48.967913 777 catalog_manager.cc:4841] TS 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275): tablet e77946b72b9e4db7a65db8bdcd8b77a7 (table test-table [id=76491abbf7a94c1cb0c0c6c36d026c45]) successfully deleted
W20260806 16:14:48.969417 778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:34443 (error 108)
I20260806 16:14:48.969468 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 985
I20260806 16:14:48.974224 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:48.973867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34443 (local address 127.0.0.1:34116)
0806 16:14:48.974036 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:48.974121 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 16:14:48.974397 778 catalog_manager.cc:4584] TS e8caa456eb0c4dbb971b02ba5ef8c31b (127.18.90.1:34443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e77946b72b9e4db7a65db8bdcd8b77a7: Network error: Client connection negotiation failed: client connection to 127.18.90.1:34443: connect: Connection refused (error 111)
I20260806 16:14:48.976019 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 1124
I20260806 16:14:48.982285 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 758
I20260806 16:14:48.989529 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38827
--webserver_interface=127.18.90.62
--webserver_port=44499
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:49.102936 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:49.103219 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:49.103261 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:49.107723 1293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:14:49.107782 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:49.107802 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:49.107820 1293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:14:49.107870 1293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:14:49.113206 1293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:38827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=44499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:49.114292 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:49.115341 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:14:49.124022 1293 server_base.cc:1034] running on GCE node
W20260806 16:14:49.123978 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:49.123973 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:49.123973 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:49.124857 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:49.125984 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:49.127177 1293 hybrid_clock.cc:648] HybridClock initialized: now 1786032889127158 us; error 29 us; skew 500 ppm
Aug 06 16:14:49 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032889, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:49.130028 1293 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:14:49.130920 1293 webserver.cc:460] Webserver started at http://127.18.90.62:44499/ using document root <none> and password file <none>
I20260806 16:14:49.131465 1293 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:49.131524 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:49.135334 1293 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:49.137463 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.138844 1293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:14:49.138988 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "d4763667d0c346918de89d738fa682dd"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "ca06b48a362b76b0b4933e843482c111"
server_key_iv: "c249f4345ff80707ab98f3e1c723cf9e"
server_key_version: "encryptionkey@0"
I20260806 16:14:49.139564 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:49.158686 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:49.161897 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:49.162103 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:49.169977 1293 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:38827
I20260806 16:14:49.170007 1360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:38827 every 8 connection(s)
I20260806 16:14:49.171093 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:14:49.175284 1361 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:14:49.176963 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1293
I20260806 16:14:49.177110 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap starting.
I20260806 16:14:49.177459 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:34443
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:38827
--webserver_port=44823
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:49.194738 1361 log.cc:824] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:49.206207 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:14:49.206666 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd: Bootstrap complete.
I20260806 16:14:49.211155 1361 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:49.211603 1361 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4763667d0c346918de89d738fa682dd, State: Initialized, Role: FOLLOWER
I20260806 16:14:49.212100 1361 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:49.212249 1361 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:14:49.212365 1361 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:14:49.212491 1361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 2 FOLLOWER]: Advancing to term 3
I20260806 16:14:49.214558 1361 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:49.214916 1361 leader_election.cc:302] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4763667d0c346918de89d738fa682dd; no voters:
I20260806 16:14:49.215272 1361 leader_election.cc:288] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 16:14:49.215379 1365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 3 FOLLOWER]: Leader election won for term 3
I20260806 16:14:49.215583 1365 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [term 3 LEADER]: Becoming Leader. State: Replica: d4763667d0c346918de89d738fa682dd, State: Running, Role: LEADER
I20260806 16:14:49.215878 1365 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } }
I20260806 16:14:49.216609 1361 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:14:49.217325 1366 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d4763667d0c346918de89d738fa682dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } } }
I20260806 16:14:49.217461 1366 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:49.217770 1367 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4763667d0c346918de89d738fa682dd. Latest consensus state: current_term: 3 leader_uuid: "d4763667d0c346918de89d738fa682dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4763667d0c346918de89d738fa682dd" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 38827 } } }
I20260806 16:14:49.217913 1367 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4763667d0c346918de89d738fa682dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:14:49.218195 1373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:14:49.224064 1373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:14:49.224879 1373 catalog_manager.cc:1259] Loaded cluster ID: 7620d98218274357b15abe30c96aa7f7
I20260806 16:14:49.224944 1373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 16:14:49.225960 1373 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff1e1a3e9ad google::LogMessage::Fail() at ??:0
@ 0x7ff1e1a4188a google::LogMessage::SendToLog() at ??:0
@ 0x7ff1e1a3e4de google::LogMessage::Flush() at ??:0
@ 0x7ff1e1a3fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff1ec381fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff1ec382d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff1ec37f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff1ec3c3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff1e3c903d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff1e3c90c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff1e3c923f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff1e3c81512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1e52c56db start_thread at ??:0
@ 0x7ff1e053c61f clone at ??:0
W20260806 16:14:49.312050 1363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:49.312426 1363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:49.312512 1363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:49.316771 1363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:49.316845 1363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:49.316931 1363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:14:49.322281 1363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:34443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=44823
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:49.323433 1363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:49.324674 1363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:49.334695 1407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:49.334846 1405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:49.334695 1404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:49.335446 1363 server_base.cc:1034] running on GCE node
I20260806 16:14:49.335870 1363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:49.336634 1363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:49.337815 1363 hybrid_clock.cc:648] HybridClock initialized: now 1786032889337796 us; error 54 us; skew 500 ppm
Aug 06 16:14:49 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032889, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:49.340580 1363 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:14:49.341498 1363 webserver.cc:460] Webserver started at http://127.18.90.1:44823/ using document root <none> and password file <none>
I20260806 16:14:49.342064 1363 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:49.342123 1363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:49.346294 1363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:14:49.348575 1414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.349901 1363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:14:49.350041 1363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b"
format_stamp: "Formatted at 2026-08-06 16:14:45 on dist-test-slave-0b58"
server_key: "f039ad6842158c3d18deb52a0bea6c4c"
server_key_iv: "f5bd5be707013b9a944fbfd9d7d68154"
server_key_version: "encryptionkey@0"
I20260806 16:14:49.350606 1363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:49.365845 1363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:49.369292 1363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:49.369509 1363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:49.370167 1363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:49.371341 1421 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:14:49.376812 1363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:14:49.376893 1363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 16:14:49.376981 1363 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:14:49.378675 1363 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:14:49.378750 1363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 16:14:49.379105 1421 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap starting.
I20260806 16:14:49.391963 1363 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:34443
I20260806 16:14:49.391986 1529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:34443 every 8 connection(s)
I20260806 16:14:49.393177 1363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:14:49.395180 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1363
I20260806 16:14:49.395684 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:36403
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:38827
--webserver_port=42945
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:14:49.403379 1421 log.cc:824] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:14:49.410678 1421 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:14:49.411065 1421 tablet_bootstrap.cc:492] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Bootstrap complete.
I20260806 16:14:49.412307 1421 ts_tablet_manager.cc:1397] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.008s
I20260806 16:14:49.414825 1421 raft_consensus.cc:348] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:49.415266 1421 raft_consensus.cc:729] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8caa456eb0c4dbb971b02ba5ef8c31b, State: Initialized, Role: FOLLOWER
I20260806 16:14:49.415762 1421 consensus_queue.cc:260] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:49.416765 1421 ts_tablet_manager.cc:1428] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260806 16:14:49.474478 1532 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:14:49.395389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:60683)
0806 16:14:49.396988 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.397050 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:49.398081 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:49.473689 (+ 75608us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1337,"thread_start_us":154,"threads_started":1}
W20260806 16:14:49.475108 1530 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:38827 (0 consecutive failures): Network error: Failed to ping master at 127.18.90.62:38827: Client connection negotiation failed: client connection to 127.18.90.62:38827: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:14:49.475889 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.475631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:41559)
0806 16:14:49.475758 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.475833 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 16:14:49.476619 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.476413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:40341)
0806 16:14:49.476519 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.476569 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 16:14:49.476816 1530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:14:49.529058 1533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:49.529333 1533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:49.529409 1533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:49.533519 1533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:49.533601 1533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:49.533697 1533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:14:49.538869 1533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:36403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=42945
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:49.539965 1533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:49.541079 1533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:49.550962 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:49.550957 1541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:49.551054 1533 server_base.cc:1034] running on GCE node
W20260806 16:14:49.550957 1544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:49.551663 1533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:49.552469 1533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:49.553642 1533 hybrid_clock.cc:648] HybridClock initialized: now 1786032889553606 us; error 49 us; skew 500 ppm
Aug 06 16:14:49 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032889, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:49.556562 1533 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:14:49.557406 1533 webserver.cc:460] Webserver started at http://127.18.90.2:42945/ using document root <none> and password file <none>
I20260806 16:14:49.557929 1533 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:49.557989 1533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:49.561810 1533 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 16:14:49.563817 1551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.565124 1533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:49.565281 1533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "51c2469d28a84eb39e1573f6cf8d8d4a"
format_stamp: "Formatted at 2026-08-06 16:14:46 on dist-test-slave-0b58"
server_key: "ae860c04e54c43c6635daea7508b44df"
server_key_iv: "26735b37814bcaa372e5881b42986fd3"
server_key_version: "encryptionkey@0"
I20260806 16:14:49.565832 1533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:49.591012 1533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:49.594139 1533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:49.594379 1533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:49.595026 1533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:49.595899 1533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:49.595974 1533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.596048 1533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:49.596094 1533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.605062 1533 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:36403
I20260806 16:14:49.605140 1665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:36403 every 8 connection(s)
I20260806 16:14:49.606125 1533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:14:49.610361 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.608500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:38275)
0806 16:14:49.608971 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.609095 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
W20260806 16:14:49.610826 1666 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:38827 (0 consecutive failures): Network error: Failed to ping master at 127.18.90.62:38827: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
I20260806 16:14:49.611441 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.611226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:47065)
0806 16:14:49.611336 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.611386 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 16:14:49.612030 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.611847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:41503)
0806 16:14:49.611941 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.611982 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 16:14:49.612195 1666 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:14:49.616250 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1533
I20260806 16:14:49.616667 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:33275
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:38827
--webserver_port=46183
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:14:49.728353 1669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:14:49.728637 1669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:14:49.728729 1669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:14:49.732740 1669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:14:49.732818 1669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:14:49.732908 1669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:14:49.738003 1669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:33275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=46183
--tserver_master_addrs=127.18.90.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:14:49.739050 1669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:14:49.740057 1669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:14:49.749542 1674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:49.749629 1669 server_base.cc:1034] running on GCE node
W20260806 16:14:49.749542 1677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:14:49.749579 1675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:14:49.750134 1669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:14:49.750728 1669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:14:49.751892 1669 hybrid_clock.cc:648] HybridClock initialized: now 1786032889751855 us; error 51 us; skew 500 ppm
Aug 06 16:14:49 dist-test-slave-0b58 krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032889, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:14:49.757956 1669 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:14:49.758878 1669 webserver.cc:460] Webserver started at http://127.18.90.3:46183/ using document root <none> and password file <none>
I20260806 16:14:49.759485 1669 fs_manager.cc:359] Metadata directory not provided
I20260806 16:14:49.759584 1669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:14:49.763541 1669 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:14:49.765820 1684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.767061 1669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:14:49.767206 1669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "1e8fa179d620489ebfe0ba8017563cb8"
format_stamp: "Formatted at 2026-08-06 16:14:47 on dist-test-slave-0b58"
server_key: "f270ea1b65ef687f2709b812e1e852ba"
server_key_iv: "d42c72b1f3e8fe6ad9aa68b5e6b25c4e"
server_key_version: "encryptionkey@0"
I20260806 16:14:49.767793 1669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:14:49.784439 1669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:14:49.787999 1669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:14:49.788152 1669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:14:49.788715 1669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:14:49.789464 1669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:14:49.789512 1669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.789551 1669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:14:49.789564 1669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:14:49.797909 1669 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:33275
I20260806 16:14:49.797977 1799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:33275 every 8 connection(s)
I20260806 16:14:49.798950 1669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:14:49.802806 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.801089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:45933)
0806 16:14:49.801537 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.801650 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
W20260806 16:14:49.803249 1800 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:38827 (0 consecutive failures): Network error: Failed to ping master at 127.18.90.62:38827: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
I20260806 16:14:49.803886 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.803624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:47111)
0806 16:14:49.803763 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.803821 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 16:14:49.804024 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1669
I20260806 16:14:49.804608 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:49.804365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:33917)
0806 16:14:49.804484 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:49.804538 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 16:14:49.804796 1800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:14:50.477877 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.477401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:44071)
0806 16:14:50.477714 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:50.477773 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260806 16:14:50.574329 1805 raft_consensus.cc:482] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:50.574525 1805 raft_consensus.cc:504] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:50.575636 1805 leader_election.cc:288] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275), 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
Aug 06 16:14:50 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032889, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.3@KRBTEST.COM
Aug 06 16:14:50 dist-test-slave-0b58 krb5kdc[32672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032889, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.2@KRBTEST.COM
I20260806 16:14:50.589524 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.575900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:36403 (local address 127.18.90.1:33419)
0806 16:14:50.576372 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:50.576388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:50.576483 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:50.578708 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:50.578710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:50.578891 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:50.579093 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:50.579108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.580472 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.580477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:50.581449 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:50.581459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.582576 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.582579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:50.582639 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:50.583231 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:50.583246 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:50.585016 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:50.587019 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:50.587025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:50.587325 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:50.587669 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:50.587675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:50.587826 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:50.588523 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:50.588530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:50.588767 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 16:14:50.588956 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 16:14:50.589318 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":97,"spinlock_wait_cycles":11264,"thread_start_us":84,"threads_started":1}
I20260806 16:14:50.589536 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.576166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:33419 (local address 127.18.90.2:36403)
0806 16:14:50.576561 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:50.576575 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:50.576611 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:14:50.577349 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:50.577356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:50.577837 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:50.578196 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:50.579287 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.580347 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.581575 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.582278 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.582431 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:50.585204 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:50.585208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:50.585219 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:50.585248 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:50.586899 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587445 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:50.587453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:50.587463 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:50.587570 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587931 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:50.587936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:50.587938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:50.588406 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:50.588548 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:50.589079 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 16:14:50.589326 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":159,"threads_started":1}
I20260806 16:14:50.589546 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.576024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:47791 (local address 127.18.90.3:33275)
0806 16:14:50.576328 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 16:14:50.576346 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:14:50.576409 (+ 63us) server_negotiation.cc:407] Connection header received
0806 16:14:50.577323 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:14:50.577332 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:14:50.577837 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:14:50.578196 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:14:50.579283 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.580323 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.581602 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.582218 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.582818 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:50.584840 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:14:50.584846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:14:50.584865 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:14:50.584892 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:14:50.586838 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587440 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:50.587447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:50.587458 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:50.587571 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587937 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:14:50.587942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:14:50.587943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:14:50.588405 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:14:50.588549 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:14:50.589064 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 16:14:50.589353 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260806 16:14:50.589553 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.575813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:33275 (local address 127.18.90.1:47791)
0806 16:14:50.575970 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:50.575988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:14:50.576101 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:14:50.578249 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:14:50.578254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:14:50.578628 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:14:50.579093 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:50.579107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.580474 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.580477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:50.581429 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:14:50.581440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:14:50.582372 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:14:50.582375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:14:50.582508 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:14:50.583093 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:14:50.583109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:14:50.584587 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:14:50.586977 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:50.586983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:50.586996 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:50.587321 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:50.587674 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:14:50.587677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:14:50.587679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:14:50.587837 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:14:50.588523 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:14:50.588530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:14:50.588767 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 16:14:50.588955 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 16:14:50.589404 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":255,"spinlock_wait_cycles":26880}
W20260806 16:14:50.592643 1416 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
W20260806 16:14:50.592836 1417 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:14:50.592947 1417 leader_election.cc:302] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8caa456eb0c4dbb971b02ba5ef8c31b; no voters: 1e8fa179d620489ebfe0ba8017563cb8, 51c2469d28a84eb39e1573f6cf8d8d4a
I20260806 16:14:50.593405 1805 raft_consensus.cc:2726] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:14:50.613128 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.612729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:57259)
0806 16:14:50.612980 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:50.613033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260806 16:14:50.805682 1810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:50.805290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:42823)
0806 16:14:50.805538 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:50.805589 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260806 16:14:51.479421 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:51.478854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:33479)
0806 16:14:51.479249 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:51.479317 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":84,"threads_started":1}
I20260806 16:14:51.614225 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:51.613820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:40577)
0806 16:14:51.614063 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:51.614116 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260806 16:14:51.806882 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:51.806426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:44495)
0806 16:14:51.806698 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:51.806760 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260806 16:14:52.480928 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:52.480493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:33417)
0806 16:14:52.480765 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:52.480827 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260806 16:14:52.615510 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:52.615067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:38567)
0806 16:14:52.615351 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:52.615411 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260806 16:14:52.796360 1816 raft_consensus.cc:482] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:52.796501 1816 raft_consensus.cc:504] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:52.796892 1816 leader_election.cc:288] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275), 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
W20260806 16:14:52.797935 1416 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
W20260806 16:14:52.798048 1417 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:14:52.798157 1417 leader_election.cc:302] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8caa456eb0c4dbb971b02ba5ef8c31b; no voters: 1e8fa179d620489ebfe0ba8017563cb8, 51c2469d28a84eb39e1573f6cf8d8d4a
I20260806 16:14:52.798337 1816 raft_consensus.cc:2726] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:14:52.808059 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:52.807618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:38227)
0806 16:14:52.807913 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:52.807973 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260806 16:14:53.482084 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:53.481660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:56101)
0806 16:14:53.481930 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:53.481985 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
I20260806 16:14:53.616747 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:53.616348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:53843)
0806 16:14:53.616593 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:53.616652 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260806 16:14:53.809162 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:53.808774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:35205)
0806 16:14:53.809017 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:53.809068 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260806 16:14:54.483403 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:54.482950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:38303)
0806 16:14:54.483251 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:54.483306 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260806 16:14:54.617939 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:54.617525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:40789)
0806 16:14:54.617774 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:54.617840 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260806 16:14:54.810281 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:54.809890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:32899)
0806 16:14:54.810132 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:54.810184 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260806 16:14:55.484791 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:55.484238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:35177)
0806 16:14:55.484625 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:55.484688 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
I20260806 16:14:55.610818 1825 raft_consensus.cc:482] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:14:55.610957 1825 raft_consensus.cc:504] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:14:55.611553 1825 leader_election.cc:288] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275), 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
W20260806 16:14:55.612351 1417 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
W20260806 16:14:55.612459 1416 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:14:55.612540 1416 leader_election.cc:302] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8caa456eb0c4dbb971b02ba5ef8c31b; no voters: 1e8fa179d620489ebfe0ba8017563cb8, 51c2469d28a84eb39e1573f6cf8d8d4a
I20260806 16:14:55.612682 1825 raft_consensus.cc:2726] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:14:55.619007 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:55.618633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:58711)
0806 16:14:55.618868 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:55.618917 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260806 16:14:55.811517 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:55.811059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:50495)
0806 16:14:55.811348 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:55.811415 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260806 16:14:56.486095 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:56.485667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:50913)
0806 16:14:56.485930 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:56.485993 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260806 16:14:56.620440 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:56.619813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:50133)
0806 16:14:56.620115 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:56.620176 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":69,"threads_started":1}
I20260806 16:14:56.812888 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:56.812392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:60835)
0806 16:14:56.812715 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:56.812773 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
I20260806 16:14:57.487353 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:57.486912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:47357)
0806 16:14:57.487191 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:57.487252 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260806 16:14:57.621654 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:57.621222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:41863)
0806 16:14:57.621496 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:57.621555 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260806 16:14:57.814095 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:57.813674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:56257)
0806 16:14:57.813945 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:57.814001 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260806 16:14:58.488601 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:58.488153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:52987)
0806 16:14:58.488455 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:58.488508 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":146,"threads_started":1}
I20260806 16:14:58.622900 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:58.622455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:53577)
0806 16:14:58.622740 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:58.622798 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260806 16:14:58.815517 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:58.814942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:37125)
0806 16:14:58.815283 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:58.815379 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260806 16:14:59.489918 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:59.489485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:42391)
0806 16:14:59.489752 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:59.489820 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260806 16:14:59.624047 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:59.623632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:41691)
0806 16:14:59.623892 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:59.623948 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260806 16:14:59.816969 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:14:59.816523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:33451)
0806 16:14:59.816808 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 16:14:59.816867 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260806 16:15:00.293939 1840 raft_consensus.cc:482] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:00.294083 1840 raft_consensus.cc:504] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8fa179d620489ebfe0ba8017563cb8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 33275 } } peers { permanent_uuid: "51c2469d28a84eb39e1573f6cf8d8d4a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 36403 } } peers { permanent_uuid: "e8caa456eb0c4dbb971b02ba5ef8c31b" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34443 } }
I20260806 16:15:00.294530 1840 leader_election.cc:288] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275), 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403)
W20260806 16:15:00.295322 1417 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e8fa179d620489ebfe0ba8017563cb8 (127.18.90.3:33275): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
W20260806 16:15:00.295473 1416 leader_election.cc:341] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51c2469d28a84eb39e1573f6cf8d8d4a (127.18.90.2:36403): Not found: Tablet not found: e77946b72b9e4db7a65db8bdcd8b77a7
I20260806 16:15:00.295552 1416 leader_election.cc:302] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8caa456eb0c4dbb971b02ba5ef8c31b; no voters: 1e8fa179d620489ebfe0ba8017563cb8, 51c2469d28a84eb39e1573f6cf8d8d4a
I20260806 16:15:00.295734 1840 raft_consensus.cc:2726] T e77946b72b9e4db7a65db8bdcd8b77a7 P e8caa456eb0c4dbb971b02ba5ef8c31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:15:00.491178 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:00.490719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:39483)
0806 16:15:00.491012 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:00.491074 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260806 16:15:00.625337 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:00.624885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:41209)
0806 16:15:00.625171 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:00.625231 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
I20260806 16:15:00.818164 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:00.817744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:49467)
0806 16:15:00.818010 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:00.818065 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260806 16:15:01.492573 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:01.492027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:40541)
0806 16:15:01.492384 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:01.492448 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":77,"threads_started":1}
I20260806 16:15:01.626806 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:01.626327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:43609)
0806 16:15:01.626625 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:01.626692 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260806 16:15:01.819507 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:01.819045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:55417)
0806 16:15:01.819346 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:01.819410 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260806 16:15:02.494029 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:02.493533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:53829)
0806 16:15:02.493839 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:02.493905 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260806 16:15:02.628101 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:02.627653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:56075)
0806 16:15:02.627920 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:02.627978 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260806 16:15:02.820824 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:02.820430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:33709)
0806 16:15:02.820667 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:02.820720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260806 16:15:03.495208 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:03.494798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:37171)
0806 16:15:03.495057 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:03.495113 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260806 16:15:03.629318 1851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:03.628858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:47323)
0806 16:15:03.629157 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:03.629216 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260806 16:15:03.821942 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:03.821547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:38925)
0806 16:15:03.821793 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:03.821847 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260806 16:15:04.496671 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:04.496136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.1:53915)
0806 16:15:04.496462 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:04.496531 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260806 16:15:04.630585 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:04.630198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.2:43185)
0806 16:15:04.630431 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:04.630485 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260806 16:15:04.807751 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 1363
I20260806 16:15:04.818382 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 1533
I20260806 16:15:04.823319 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:04.822897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:38827 (local address 127.18.90.3:42587)
0806 16:15:04.823144 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:04.823204 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260806 16:15:04.825012 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 1669
2026-08-06T16:15:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20463 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:15:04 dist-test-slave-0b58 krb5kdc[1858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:15:04 dist-test-slave-0b58 krb5kdc[1858](info): set up 2 sockets
Aug 06 16:15:04 dist-test-slave-0b58 krb5kdc[1858](info): commencing operation
krb5kdc: starting...
W20260806 16:15:05.852711 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:15:05 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:15:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:15:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:15:05.977003 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36931
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:06.092454 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:06.092758 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:06.092798 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:06.096788 1871 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 16:15:06.096865 1871 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 16:15:06.096881 1871 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 16:15:06.096895 1871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:15:06.096915 1871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:06.096932 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:06.096947 1871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:15:06.096962 1871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:15:06.102371 1871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:06.103495 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:06.104573 1871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:06.113157 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:06.113097 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:06.113101 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:06.113384 1871 server_base.cc:1034] running on GCE node
I20260806 16:15:06.114032 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:06.115280 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:06.116467 1871 hybrid_clock.cc:648] HybridClock initialized: now 1786032906116448 us; error 35 us; skew 500 ppm
Aug 06 16:15:06 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:06.123929 1871 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:15:06.125042 1871 webserver.cc:460] Webserver started at http://127.18.90.62:34683/ using document root <none> and password file <none>
I20260806 16:15:06.125888 1871 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:06.126003 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:06.126279 1871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:15:06.128549 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "5d2f777eda55d0a409a43e659a718d67"
server_key_iv: "e87d12db650776ebf3a20c1f0b48969f"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.129158 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "5d2f777eda55d0a409a43e659a718d67"
server_key_iv: "e87d12db650776ebf3a20c1f0b48969f"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.133186 1871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:15:06.135749 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.137149 1871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:15:06.137318 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "5d2f777eda55d0a409a43e659a718d67"
server_key_iv: "e87d12db650776ebf3a20c1f0b48969f"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.137444 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:06.153062 1871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:06.156364 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:06.156565 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:06.163513 1871 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:36931
I20260806 16:15:06.163811 1939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:36931 every 8 connection(s)
I20260806 16:15:06.164569 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:15:06.167377 1940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:06.173063 1940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap starting.
I20260806 16:15:06.174105 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1871
I20260806 16:15:06.174247 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:15:06.174521 18792 external_mini_cluster.cc:1428] Setting key 77055d54f07ffa8e238e144fb05ba74d
I20260806 16:15:06.175990 1940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:06.176812 1940 log.cc:824] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:06.178740 1940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: No bootstrap required, opened a new log
I20260806 16:15:06.182215 1940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:06.182463 1940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:06.182524 1940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc362ab9ae74d27ac37c5b5a1d5b1aa, State: Initialized, Role: FOLLOWER
I20260806 16:15:06.182953 1940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
Aug 06 16:15:06 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:06.183070 1940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:15:06.183192 1940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:15:06.183305 1940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:15:06.184221 1940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:06.184584 1940 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc362ab9ae74d27ac37c5b5a1d5b1aa; no voters:
I20260806 16:15:06.184957 1940 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:15:06.185065 1945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:15:06.185299 1945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 LEADER]: Becoming Leader. State: Replica: 7bc362ab9ae74d27ac37c5b5a1d5b1aa, State: Running, Role: LEADER
I20260806 16:15:06.185649 1945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:06.186388 1940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:15:06.187604 1946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } } }
I20260806 16:15:06.187729 1946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: This master's current role is: LEADER
I20260806 16:15:06.187992 1947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc362ab9ae74d27ac37c5b5a1d5b1aa. Latest consensus state: current_term: 1 leader_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } } }
I20260806 16:15:06.188082 1947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: This master's current role is: LEADER
I20260806 16:15:06.189953 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:06.175666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.18.90.62:36931)
0806 16:15:06.176118 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:06.176139 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:06.176181 (+ 42us) server_negotiation.cc:407] Connection header received
0806 16:15:06.176844 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:06.176861 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:06.177237 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:06.177579 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:06.178387 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.179468 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.180376 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.180866 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.181047 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:06.183662 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:06.183667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:06.183681 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:06.183707 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:06.185879 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:06.186406 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:06.186413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:06.186420 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:06.186569 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:06.186883 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:06.186886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:06.186887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:06.187436 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:06.188401 (+ 965us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:06.188555 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 16:15:06.188872 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
W20260806 16:15:06.191395 1960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:15:06.191473 1960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:15:06.191533 1954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:15:06.192516 1954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:15:06.197809 1954 catalog_manager.cc:1347] Generated new cluster ID: 6a20c46b81f441eabdc36d82ad1dca51
I20260806 16:15:06.197877 1954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:15:06.216357 1954 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:15:06.217394 1954 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:15:06.222393 1954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Generated new TSK 0
I20260806 16:15:06.223385 1954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:15:06.257310 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36931
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:06.448244 1966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:06.448807 1966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:06.448876 1966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:06.454968 1966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:06.455070 1966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:06.455164 1966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:15:06.463467 1966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:06.465452 1966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:06.467144 1966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:06.483367 1971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:06.483402 1974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:06.483398 1972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:06.483819 1966 server_base.cc:1034] running on GCE node
I20260806 16:15:06.484844 1966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:06.485973 1966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:06.487289 1966 hybrid_clock.cc:648] HybridClock initialized: now 1786032906487252 us; error 50 us; skew 500 ppm
Aug 06 16:15:06 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:06.492065 1966 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:15:06.493579 1966 webserver.cc:460] Webserver started at http://127.18.90.1:34257/ using document root <none> and password file <none>
I20260806 16:15:06.494508 1966 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:06.494625 1966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:06.494967 1966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:15:06.498531 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "bcad3d88247d4050860cfaf6808a8189"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "64fd017ab4f36eefef4d4350b8c0cd2d"
server_key_iv: "bc01279c0e02cb1075aa879e41631c6c"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.499583 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "bcad3d88247d4050860cfaf6808a8189"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "64fd017ab4f36eefef4d4350b8c0cd2d"
server_key_iv: "bc01279c0e02cb1075aa879e41631c6c"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.506759 1966 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 16:15:06.511053 1981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.513490 1966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:15:06.513753 1966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "bcad3d88247d4050860cfaf6808a8189"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "64fd017ab4f36eefef4d4350b8c0cd2d"
server_key_iv: "bc01279c0e02cb1075aa879e41631c6c"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.513921 1966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:06.531900 1966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:06.537537 1966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:06.537887 1966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:06.538868 1966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:06.540341 1966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:06.540419 1966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.540534 1966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:06.540608 1966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.557175 1966 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:37061
I20260806 16:15:06.557219 2097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:37061 every 8 connection(s)
I20260806 16:15:06.558874 1966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:15:06.567184 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 1966
I20260806 16:15:06.567377 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:15:06.567857 18792 external_mini_cluster.cc:1428] Setting key 4ed72b509ed944c5c567697a92eae707
Aug 06 16:15:06 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:06.585666 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:06.561843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:45223 (local address 127.18.90.62:36931)
0806 16:15:06.562043 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:06.562048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:06.563523 (+ 1475us) server_negotiation.cc:407] Connection header received
0806 16:15:06.566099 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:06.566104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:06.566218 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:06.566391 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:06.568836 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.569815 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.572006 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.572524 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.572778 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:06.577946 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:06.577951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:06.577967 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:06.578028 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:06.581276 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:06.582616 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:06.582619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:06.582621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:06.582764 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:06.583511 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:06.583514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:06.583515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:06.583837 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:06.583961 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:06.585412 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0806 16:15:06.585535 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:15:06.588135 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:06.562575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:45223)
0806 16:15:06.563292 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:06.563360 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:06.565759 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:06.566677 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:06.566694 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:06.567449 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:06.568535 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:06.568582 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.570026 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.570030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:06.571787 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:06.571805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.572828 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.572834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:06.573085 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:06.574045 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:06.574084 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:06.577447 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:06.581506 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:06.581529 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:06.581574 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:06.582397 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:06.582925 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:06.582936 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:06.582938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:06.583355 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:06.584009 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:06.584020 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:06.584891 (+ 871us) client_negotiation.cc:769] Sending connection context
0806 16:15:06.585351 (+ 460us) client_negotiation.cc:240] Negotiation successful
0806 16:15:06.585856 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":208,"threads_started":1}
I20260806 16:15:06.590358 2098 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:15:06.590763 2098 heartbeater.cc:464] Registering TS with master...
I20260806 16:15:06.591555 2098 heartbeater.cc:511] Master 127.18.90.62:36931 requested a full tablet report, sending...
I20260806 16:15:06.594139 1904 ts_manager.cc:194] Registered new tserver with Master: bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:06.596673 1904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:45223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:15:06.615444 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36931
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:06.823441 2103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:06.823930 2103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:06.823997 2103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:06.830773 2103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:06.830868 2103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:06.830951 2103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:15:06.840161 2103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:06.842525 2103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:06.844486 2103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:06.860396 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:06.860617 2103 server_base.cc:1034] running on GCE node
W20260806 16:15:06.860416 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:06.861063 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:06.861696 2103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:06.862707 2103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:06.863989 2103 hybrid_clock.cc:648] HybridClock initialized: now 1786032906863949 us; error 55 us; skew 500 ppm
Aug 06 16:15:06 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:06.868978 2103 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:15:06.870429 2103 webserver.cc:460] Webserver started at http://127.18.90.2:37805/ using document root <none> and password file <none>
I20260806 16:15:06.871398 2103 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:06.871524 2103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:06.871925 2103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:15:06.875547 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "acdf2b6e506a4948933cfc43645936f1"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "35c38ee47f87d17a3f916a539128c890"
server_key_iv: "f1b8f9d0025cb904f12e6f6b89b8f87a"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.876555 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "acdf2b6e506a4948933cfc43645936f1"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "35c38ee47f87d17a3f916a539128c890"
server_key_iv: "f1b8f9d0025cb904f12e6f6b89b8f87a"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.882911 2103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 16:15:06.887176 2118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.889335 2103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:15:06.889564 2103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "acdf2b6e506a4948933cfc43645936f1"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "35c38ee47f87d17a3f916a539128c890"
server_key_iv: "f1b8f9d0025cb904f12e6f6b89b8f87a"
server_key_version: "encryptionkey@0"
I20260806 16:15:06.889720 2103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:06.970055 2103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:06.973208 2103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:06.973433 2103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:06.974083 2103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:06.974939 2103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:06.975008 2103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.975080 2103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:06.975121 2103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:06.984467 2103 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44317
I20260806 16:15:06.984498 2233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44317 every 8 connection(s)
I20260806 16:15:06.985435 2103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:15:06.989346 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2103
I20260806 16:15:06.989439 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:15:06.989754 18792 external_mini_cluster.cc:1428] Setting key 1fe9a4ce55adfb5015bb4079bb02e2ba
Aug 06 16:15:06 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:07.000131 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:06.987215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:55803 (local address 127.18.90.62:36931)
0806 16:15:06.987379 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:06.987384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:06.988242 (+ 858us) server_negotiation.cc:407] Connection header received
0806 16:15:06.989736 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:06.989740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:06.989810 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:06.989907 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:06.991389 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.991933 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.993370 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.993586 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.993689 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:06.996180 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:06.996184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:06.996187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:06.996216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:06.997896 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:06.998564 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:06.998568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:06.998570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:06.998658 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:06.999000 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:06.999004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:06.999006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:06.999198 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:06.999307 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:06.999834 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 16:15:06.999960 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:15:07.001020 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:06.987630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:55803)
0806 16:15:06.988084 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:06.988128 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:06.989482 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:06.990095 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:06.990105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:06.990593 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:06.991235 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:06.991254 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.992297 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.992313 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:06.993244 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:06.993254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:06.993739 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:06.993743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:06.993916 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:06.994496 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:06.994511 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:06.996017 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:06.998039 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:06.998045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:06.998061 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:06.998456 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:06.998756 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:06.998759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:06.998761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:06.998901 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:06.999321 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:06.999329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:06.999579 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 16:15:06.999775 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.000124 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260806 16:15:07.002241 2235 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36931
I20260806 16:15:07.002509 2235 heartbeater.cc:464] Registering TS with master...
I20260806 16:15:07.003078 2235 heartbeater.cc:511] Master 127.18.90.62:36931 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:15:07.004179 1904 ts_manager.cc:194] Registered new tserver with Master: acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
I20260806 16:15:07.004854 1904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:55803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:15:07.019660 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36931
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:07.191735 2240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:07.192301 2240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:07.192476 2240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:07.198537 2240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:07.198634 2240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:07.198824 2240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:15:07.207051 2240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:07.208947 2240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:07.210716 2240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:07.227545 2245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:07.227488 2248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:07.227715 2240 server_base.cc:1034] running on GCE node
W20260806 16:15:07.227555 2246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:07.228679 2240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:07.229586 2240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:07.230785 2240 hybrid_clock.cc:648] HybridClock initialized: now 1786032907230752 us; error 56 us; skew 500 ppm
Aug 06 16:15:07 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032907, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:07.235255 2240 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:15:07.236640 2240 webserver.cc:460] Webserver started at http://127.18.90.3:36517/ using document root <none> and password file <none>
I20260806 16:15:07.237568 2240 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:07.237679 2240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:07.237993 2240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:15:07.241338 2240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "b961a4432c224396b18946906acec2c8"
format_stamp: "Formatted at 2026-08-06 16:15:07 on dist-test-slave-0b58"
server_key: "4eb92470939d05c2f22ff7b4a140bf24"
server_key_iv: "1caeba0104602c63cca50d4b3992dbe1"
server_key_version: "encryptionkey@0"
I20260806 16:15:07.242324 2240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "b961a4432c224396b18946906acec2c8"
format_stamp: "Formatted at 2026-08-06 16:15:07 on dist-test-slave-0b58"
server_key: "4eb92470939d05c2f22ff7b4a140bf24"
server_key_iv: "1caeba0104602c63cca50d4b3992dbe1"
server_key_version: "encryptionkey@0"
I20260806 16:15:07.248440 2240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 16:15:07.252190 2255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:07.254173 2240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:15:07.254338 2240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "b961a4432c224396b18946906acec2c8"
format_stamp: "Formatted at 2026-08-06 16:15:07 on dist-test-slave-0b58"
server_key: "4eb92470939d05c2f22ff7b4a140bf24"
server_key_iv: "1caeba0104602c63cca50d4b3992dbe1"
server_key_version: "encryptionkey@0"
I20260806 16:15:07.254484 2240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:07.281636 2240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:07.286379 2240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:07.286597 2240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:07.287710 2240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:07.289330 2240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:07.289458 2240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:07.289597 2240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:07.289659 2240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:07.304565 2240 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:41911
I20260806 16:15:07.304643 2371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:41911 every 8 connection(s)
I20260806 16:15:07.306080 2240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:15:07.310088 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2240
I20260806 16:15:07.310278 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:15:07.310858 18792 external_mini_cluster.cc:1428] Setting key 64930e5ab9b72fe8d805dd9e8b6a950e
Aug 06 16:15:07 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032907, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:07.328691 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.308863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:33979 (local address 127.18.90.62:36931)
0806 16:15:07.309093 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.309107 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.310771 (+ 1664us) server_negotiation.cc:407] Connection header received
0806 16:15:07.313560 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.313564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.313660 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.313825 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:07.315853 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.316635 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.318382 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.318674 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.318790 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.322725 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:07.322738 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:07.322741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:07.322777 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:07.324765 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:07.326069 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:07.326076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:07.326084 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:07.326285 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:07.326956 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:07.326960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:07.326961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:07.327362 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:07.327483 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.328213 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.328478 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 16:15:07.330084 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.309616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:33979)
0806 16:15:07.310433 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.310597 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:07.313238 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:07.314178 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:07.314214 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:07.314843 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:07.315637 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.315669 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.316837 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.316841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.318193 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.318203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.318842 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.318846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.319101 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.319984 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:07.320077 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:07.322475 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:07.325038 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:07.325059 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:07.325106 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:07.325890 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:07.326458 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:07.326460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:07.326461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:07.326816 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:07.327487 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:07.327492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:07.327905 (+ 413us) client_negotiation.cc:769] Sending connection context
0806 16:15:07.328136 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.328750 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":206,"threads_started":1}
I20260806 16:15:07.331647 2372 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36931
I20260806 16:15:07.332015 2372 heartbeater.cc:464] Registering TS with master...
I20260806 16:15:07.332994 2372 heartbeater.cc:511] Master 127.18.90.62:36931 requested a full tablet report, sending...
I20260806 16:15:07.334564 1904 ts_manager.cc:194] Registered new tserver with Master: b961a4432c224396b18946906acec2c8 (127.18.90.3:41911)
I20260806 16:15:07.335374 1904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:33979
I20260806 16:15:07.339197 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:15:07.352350 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.342271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.18.90.62:36931)
0806 16:15:07.342471 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.342478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.342783 (+ 305us) server_negotiation.cc:407] Connection header received
0806 16:15:07.342923 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.342925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.342978 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.343076 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:07.344012 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.344800 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.345918 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.346187 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.346295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.348273 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:07.348277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:07.348279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:07.348326 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:07.350183 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:07.350724 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:07.350727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:07.350734 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:07.350835 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:07.351091 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:07.351093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:07.351095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:07.351387 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:07.351566 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.351827 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.351977 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:15:07.356307 1904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:15:07.360700 1904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:15:07.381724 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.373292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41911 (local address 127.0.0.1:34590)
0806 16:15:07.375018 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.375105 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:07.375340 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:07.376241 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:07.376282 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:07.376322 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:07.376541 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.376554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.378057 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.378065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.379576 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.379592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.381007 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.381011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.381151 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.381213 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 16:15:07.381377 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.381536 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"thread_start_us":173,"threads_started":1}
I20260806 16:15:07.382386 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.373611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44317 (local address 127.0.0.1:37630)
0806 16:15:07.375018 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.375102 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:07.375304 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:07.376538 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:07.376540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:07.376562 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:07.376707 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.376721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.378664 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.378669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.380195 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.380209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.381789 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.381796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.381938 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.381972 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 16:15:07.382076 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.382190 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":97,"thread_start_us":192,"threads_started":1}
I20260806 16:15:07.382661 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.373486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.18.90.3:41911)
0806 16:15:07.374601 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.374630 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.375242 (+ 612us) server_negotiation.cc:407] Connection header received
0806 16:15:07.375635 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.375644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.375942 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.376743 (+ 801us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:07.376957 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.377890 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.379764 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.380826 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.381006 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.381221 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.382253 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.382391 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":125,"threads_started":1}
I20260806 16:15:07.382753 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.374677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.0.0.1:37614)
0806 16:15:07.375103 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.375115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:07.375295 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:07.376774 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:07.376777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:07.376811 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:07.376960 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.376969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.378810 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.378815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.380247 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.380299 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.382077 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.382081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.382217 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.382265 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 16:15:07.382384 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.382500 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":5,"thread_start_us":166,"threads_started":1}
I20260806 16:15:07.383227 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.373862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.18.90.2:44317)
0806 16:15:07.374751 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.374782 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.375229 (+ 447us) server_negotiation.cc:407] Connection header received
0806 16:15:07.375625 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.375639 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.375942 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.376176 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:07.376860 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.378448 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.380425 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.381630 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.381782 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.382350 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.382432 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.383023 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":229,"threads_started":1}
I20260806 16:15:07.383364 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.374946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.18.90.1:37061)
0806 16:15:07.375831 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.375841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.375876 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:15:07.375987 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.376001 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.376382 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.376615 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:07.377156 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.378670 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.380430 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.381633 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.381777 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.381994 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.382982 (+ 988us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.383143 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":189,"threads_started":1}
I20260806 16:15:07.387368 2168 tablet_service.cc:1467] Processing CreateTablet for tablet 77e3f528463e4df8bf8d77563d33d406 (DEFAULT_TABLE table=test-table [id=385118a08c2148898570956c5884502c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:15:07.387401 2306 tablet_service.cc:1467] Processing CreateTablet for tablet 77e3f528463e4df8bf8d77563d33d406 (DEFAULT_TABLE table=test-table [id=385118a08c2148898570956c5884502c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:15:07.388319 2030 tablet_service.cc:1467] Processing CreateTablet for tablet 77e3f528463e4df8bf8d77563d33d406 (DEFAULT_TABLE table=test-table [id=385118a08c2148898570956c5884502c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:15:07.389087 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e3f528463e4df8bf8d77563d33d406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:07.389094 2306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e3f528463e4df8bf8d77563d33d406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:07.389703 2030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e3f528463e4df8bf8d77563d33d406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:07.398116 2391 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Bootstrap starting.
I20260806 16:15:07.400115 2392 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Bootstrap starting.
I20260806 16:15:07.401762 2393 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Bootstrap starting.
I20260806 16:15:07.403055 2391 tablet_bootstrap.cc:654] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:07.403735 2392 tablet_bootstrap.cc:654] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:07.404615 2393 tablet_bootstrap.cc:654] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:07.405658 2393 log.cc:824] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:07.405786 2391 log.cc:824] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:07.405838 2392 log.cc:824] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:07.408751 2391 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: No bootstrap required, opened a new log
I20260806 16:15:07.408754 2393 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: No bootstrap required, opened a new log
I20260806 16:15:07.409037 2393 ts_tablet_manager.cc:1397] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 16:15:07.409039 2391 ts_tablet_manager.cc:1397] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260806 16:15:07.411402 2392 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: No bootstrap required, opened a new log
I20260806 16:15:07.411681 2392 ts_tablet_manager.cc:1397] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260806 16:15:07.414161 2393 raft_consensus.cc:348] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.414525 2393 raft_consensus.cc:374] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:07.414613 2393 raft_consensus.cc:729] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcad3d88247d4050860cfaf6808a8189, State: Initialized, Role: FOLLOWER
I20260806 16:15:07.414868 2391 raft_consensus.cc:348] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.415370 2391 raft_consensus.cc:374] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:07.415278 2393 consensus_queue.cc:260] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.415472 2391 raft_consensus.cc:729] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acdf2b6e506a4948933cfc43645936f1, State: Initialized, Role: FOLLOWER
I20260806 16:15:07.416109 2391 consensus_queue.cc:260] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.416318 2098 heartbeater.cc:503] Master 127.18.90.62:36931 was elected leader, sending a full tablet report...
I20260806 16:15:07.416966 2393 ts_tablet_manager.cc:1428] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 16:15:07.417140 2235 heartbeater.cc:503] Master 127.18.90.62:36931 was elected leader, sending a full tablet report...
I20260806 16:15:07.417708 2391 ts_tablet_manager.cc:1428] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 16:15:07.419224 2392 raft_consensus.cc:348] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.419716 2392 raft_consensus.cc:374] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:07.419803 2392 raft_consensus.cc:729] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b961a4432c224396b18946906acec2c8, State: Initialized, Role: FOLLOWER
I20260806 16:15:07.420536 2392 consensus_queue.cc:260] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.422155 2392 ts_tablet_manager.cc:1428] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 16:15:07.422595 2372 heartbeater.cc:503] Master 127.18.90.62:36931 was elected leader, sending a full tablet report...
I20260806 16:15:07.475003 2399 raft_consensus.cc:482] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:07.475383 2399 raft_consensus.cc:504] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.477104 2399 leader_election.cc:288] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317), bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:07.481916 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.477413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:38405 (local address 127.18.90.2:44317)
0806 16:15:07.477565 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.477569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.477585 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:15:07.477776 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.477779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.477823 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.477936 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:07.478220 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.479007 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.480650 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.481236 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.481355 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.481420 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.481712 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.481785 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:15:07.481916 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.477551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:47113 (local address 127.18.90.1:37061)
0806 16:15:07.477723 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.477728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.477858 (+ 130us) server_negotiation.cc:407] Connection header received
0806 16:15:07.477975 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.477979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.478031 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.478099 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:07.478369 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.479087 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.480650 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.481277 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.481369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.481420 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.481703 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.481782 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:15:07.481948 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.477304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44317 (local address 127.18.90.3:38405)
0806 16:15:07.477525 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.477541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:07.477632 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:07.477958 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:07.477962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:07.477981 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:07.478101 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.478110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.479107 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.479109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.480526 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.480538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.481366 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.481370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.481538 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.481596 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 16:15:07.481676 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.481795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":140}
I20260806 16:15:07.481956 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.477374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.18.90.3:47113)
0806 16:15:07.477780 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.477792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:07.477865 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:07.478135 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:07.478141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:07.478162 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:07.478265 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.478269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.479217 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.479220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.480534 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:07.480539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.481370 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.481373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:07.481546 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.481601 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 16:15:07.481676 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:15:07.481795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":134,"thread_start_us":141,"threads_started":1}
I20260806 16:15:07.482762 2188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3f528463e4df8bf8d77563d33d406" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdf2b6e506a4948933cfc43645936f1" is_pre_election: true
I20260806 16:15:07.482758 2051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3f528463e4df8bf8d77563d33d406" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcad3d88247d4050860cfaf6808a8189" is_pre_election: true
I20260806 16:15:07.483078 2051 raft_consensus.cc:2454] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 0.
I20260806 16:15:07.483078 2188 raft_consensus.cc:2454] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 0.
I20260806 16:15:07.483546 2256 leader_election.cc:302] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b961a4432c224396b18946906acec2c8, bcad3d88247d4050860cfaf6808a8189; no voters:
I20260806 16:15:07.483788 2399 raft_consensus.cc:2781] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:15:07.483865 2399 raft_consensus.cc:482] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:15:07.483896 2399 raft_consensus.cc:3037] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:15:07.485234 2399 raft_consensus.cc:504] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.485595 2399 leader_election.cc:288] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 election: Requested vote from peers acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317), bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:07.485980 2051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3f528463e4df8bf8d77563d33d406" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcad3d88247d4050860cfaf6808a8189"
I20260806 16:15:07.485991 2188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3f528463e4df8bf8d77563d33d406" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdf2b6e506a4948933cfc43645936f1"
I20260806 16:15:07.486142 2188 raft_consensus.cc:3037] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:15:07.486100 2051 raft_consensus.cc:3037] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Advancing to term 1
W20260806 16:15:07.486617 2236 tablet.cc:2369] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:15:07.487434 2188 raft_consensus.cc:2454] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 1.
I20260806 16:15:07.487746 2051 raft_consensus.cc:2454] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 1.
I20260806 16:15:07.487857 2257 leader_election.cc:302] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8; no voters:
I20260806 16:15:07.488008 2399 raft_consensus.cc:2781] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:15:07.488296 2399 raft_consensus.cc:686] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 1 LEADER]: Becoming Leader. State: Replica: b961a4432c224396b18946906acec2c8, State: Running, Role: LEADER
I20260806 16:15:07.488636 2399 consensus_queue.cc:237] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:07.491879 1903 catalog_manager.cc:5495] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b961a4432c224396b18946906acec2c8 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "b961a4432c224396b18946906acec2c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } health_report { overall_health: UNKNOWN } } }
I20260806 16:15:07.503340 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.499680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.18.90.3:41911)
0806 16:15:07.499837 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:07.499842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:07.499857 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:15:07.499995 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:07.499998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:07.500049 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:07.500148 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:15:07.500491 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.500984 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.501851 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:07.502071 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:07.502180 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:07.502496 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:15:07.502940 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:15:07.503068 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:07.503126 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 16:15:07.503181 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:15:07.509207 2188 raft_consensus.cc:1261] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Refusing update from remote peer b961a4432c224396b18946906acec2c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:15:07.509213 2051 raft_consensus.cc:1261] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Refusing update from remote peer b961a4432c224396b18946906acec2c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:15:07.509924 2399 consensus_queue.cc:1035] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:15:07.510110 2401 consensus_queue.cc:1035] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:15:07.518033 2405 mvcc.cc:205] Tried to move back new op lower bound from 7315590789150031872 to 7315590789076103168. Current Snapshot: MvccSnapshot[applied={T|T < 7315590789150031872}]
I20260806 16:15:07.518983 2406 mvcc.cc:205] Tried to move back new op lower bound from 7315590789150031872 to 7315590789076103168. Current Snapshot: MvccSnapshot[applied={T|T < 7315590789150031872}]
I20260806 16:15:07.527233 1903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:15:07.527415 1903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:15:07.529832 1903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Sending DeleteTablet for 3 replicas of tablet 77e3f528463e4df8bf8d77563d33d406
I20260806 16:15:07.530568 2306 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3f528463e4df8bf8d77563d33d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:34590
I20260806 16:15:07.530696 2168 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3f528463e4df8bf8d77563d33d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:37630
I20260806 16:15:07.530771 2030 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3f528463e4df8bf8d77563d33d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:37614
I20260806 16:15:07.531265 2416 tablet_replica.cc:330] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: stopping tablet replica
I20260806 16:15:07.531363 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 1966
I20260806 16:15:07.531679 2416 raft_consensus.cc:2229] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:15:07.532025 2416 raft_consensus.cc:2258] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:15:07.532532 2415 tablet_replica.cc:330] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: stopping tablet replica
I20260806 16:15:07.532953 2415 raft_consensus.cc:2229] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:15:07.533423 2415 raft_consensus.cc:2258] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:15:07.534493 2416 ts_tablet_manager.cc:1905] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:15:07.535867 2415 ts_tablet_manager.cc:1905] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:15:07.537716 2416 ts_tablet_manager.cc:1918] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:15:07.537894 2416 log.cc:1196] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/77e3f528463e4df8bf8d77563d33d406
I20260806 16:15:07.538316 2416 ts_tablet_manager.cc:1939] T 77e3f528463e4df8bf8d77563d33d406 P acdf2b6e506a4948933cfc43645936f1: Deleting consensus metadata
I20260806 16:15:07.539686 1888 catalog_manager.cc:4841] TS acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): tablet 77e3f528463e4df8bf8d77563d33d406 (table test-table [id=385118a08c2148898570956c5884502c]) successfully deleted
I20260806 16:15:07.540331 2415 ts_tablet_manager.cc:1918] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:15:07.540477 2415 log.cc:1196] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/77e3f528463e4df8bf8d77563d33d406
I20260806 16:15:07.541134 2415 ts_tablet_manager.cc:1939] T 77e3f528463e4df8bf8d77563d33d406 P b961a4432c224396b18946906acec2c8: Deleting consensus metadata
I20260806 16:15:07.542671 1889 catalog_manager.cc:4841] TS b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): tablet 77e3f528463e4df8bf8d77563d33d406 (table test-table [id=385118a08c2148898570956c5884502c]) successfully deleted
W20260806 16:15:07.549996 1887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:37061 (error 108)
I20260806 16:15:07.551214 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 2103
I20260806 16:15:07.551347 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:07.551047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.0.0.1:37624)
0806 16:15:07.551165 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:07.551240 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 16:15:07.552659 1887 catalog_manager.cc:4584] TS bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77e3f528463e4df8bf8d77563d33d406: Network error: Client connection negotiation failed: client connection to 127.18.90.1:37061: connect: Connection refused (error 111)
I20260806 16:15:07.560500 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 2240
I20260806 16:15:07.567498 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 1871
I20260806 16:15:27.583745 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36931
--webserver_interface=127.18.90.62
--webserver_port=34683
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:27.702446 2419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:27.702695 2419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:27.702751 2419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:27.706760 2419 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 16:15:27.706810 2419 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 16:15:27.706826 2419 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 16:15:27.706840 2419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:15:27.706900 2419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:27.706930 2419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:27.706970 2419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:15:27.707010 2419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:15:27.712162 2419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=34683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:27.713306 2419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:27.714341 2419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:27.727727 2428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:27.727771 2425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:27.727733 2426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:27.727936 2419 server_base.cc:1034] running on GCE node
I20260806 16:15:27.728724 2419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:27.729861 2419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:27.731055 2419 hybrid_clock.cc:648] HybridClock initialized: now 1786032927731037 us; error 32 us; skew 500 ppm
Aug 06 16:15:27 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032927, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:27.738729 2419 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:15:27.739683 2419 webserver.cc:460] Webserver started at http://127.18.90.62:34683/ using document root <none> and password file <none>
I20260806 16:15:27.740368 2419 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:27.740477 2419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:27.744491 2419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:15:27.746694 2435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:27.747957 2419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:15:27.748111 2419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "5d2f777eda55d0a409a43e659a718d67"
server_key_iv: "e87d12db650776ebf3a20c1f0b48969f"
server_key_version: "encryptionkey@0"
I20260806 16:15:27.748831 2419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:27.766314 2419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:27.769153 2419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:27.769363 2419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:27.776939 2419 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:36931
I20260806 16:15:27.777257 2488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:36931 every 8 connection(s)
I20260806 16:15:27.778083 2419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:15:27.782470 2489 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:15:27.784101 2489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap starting.
I20260806 16:15:27.784217 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2419
I20260806 16:15:27.784685 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:37061
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:36931
--webserver_port=34257
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:15:27.801790 2489 log.cc:824] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:27.810411 2489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:15:27.810801 2489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap complete.
I20260806 16:15:27.814997 2489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:27.815419 2489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc362ab9ae74d27ac37c5b5a1d5b1aa, State: Initialized, Role: FOLLOWER
I20260806 16:15:27.815893 2489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:27.816087 2489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:15:27.816175 2489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:15:27.816322 2489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:15:27.818251 2489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:27.818590 2489 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc362ab9ae74d27ac37c5b5a1d5b1aa; no voters:
I20260806 16:15:27.818900 2489 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:15:27.819064 2493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:15:27.819420 2493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 LEADER]: Becoming Leader. State: Replica: 7bc362ab9ae74d27ac37c5b5a1d5b1aa, State: Running, Role: LEADER
I20260806 16:15:27.819828 2493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:27.820103 2489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:15:27.821295 2494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } } }
I20260806 16:15:27.821410 2494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: This master's current role is: LEADER
I20260806 16:15:27.822178 2495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc362ab9ae74d27ac37c5b5a1d5b1aa. Latest consensus state: current_term: 2 leader_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } } }
I20260806 16:15:27.822340 2495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: This master's current role is: LEADER
I20260806 16:15:27.825399 2509 catalog_manager.cc:1259] Loaded cluster ID: 6a20c46b81f441eabdc36d82ad1dca51
I20260806 16:15:27.825459 2509 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: loading cluster ID for follower catalog manager: success
I20260806 16:15:27.827996 2509 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: acquiring CA information for follower catalog manager: success
I20260806 16:15:27.829655 2509 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:15:27.829851 2502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:15:27.832481 2502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:15:27.833003 2502 catalog_manager.cc:1259] Loaded cluster ID: 6a20c46b81f441eabdc36d82ad1dca51
I20260806 16:15:27.833113 2502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:15:27.833843 2502 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:15:27.834372 2502 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Loaded TSK: 0
I20260806 16:15:27.845885 2502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Generated new TSK 1
I20260806 16:15:27.846242 2502 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:15:27.903769 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:27.904057 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:27.904147 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:27.908144 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:27.908205 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:27.908309 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:15:27.913451 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:37061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=34257
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:27.914441 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:27.915462 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:27.925904 2517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:27.926059 2491 server_base.cc:1034] running on GCE node
W20260806 16:15:27.925908 2518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:27.926236 2520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:27.926774 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:27.927629 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:27.928838 2491 hybrid_clock.cc:648] HybridClock initialized: now 1786032927928805 us; error 39 us; skew 500 ppm
Aug 06 16:15:27 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032927, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:27.932237 2491 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:15:27.933210 2491 webserver.cc:460] Webserver started at http://127.18.90.1:34257/ using document root <none> and password file <none>
I20260806 16:15:27.933799 2491 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:27.933908 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:27.937955 2491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:15:27.940296 2527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:27.941702 2491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:15:27.941835 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "bcad3d88247d4050860cfaf6808a8189"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "64fd017ab4f36eefef4d4350b8c0cd2d"
server_key_iv: "bc01279c0e02cb1075aa879e41631c6c"
server_key_version: "encryptionkey@0"
I20260806 16:15:27.942394 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:27.965332 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:27.968226 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:27.968487 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:27.969090 2491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:27.970170 2534 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:15:27.974120 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:15:27.974210 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260806 16:15:27.974304 2491 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:15:27.975932 2491 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:15:27.976001 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 16:15:27.976238 2534 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Bootstrap starting.
I20260806 16:15:27.986370 2491 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:37061
I20260806 16:15:27.986413 2643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:37061 every 8 connection(s)
I20260806 16:15:27.987381 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:15:27.992097 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2491
I20260806 16:15:27.992663 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:44317
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:36931
--webserver_port=37805
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:15:27.995239 2534 log.cc:824] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:15:27 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032927, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:28.003569 2534 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:15:28.003976 2534 tablet_bootstrap.cc:492] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Bootstrap complete.
I20260806 16:15:28.005422 2534 ts_tablet_manager.cc:1397] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.007s
I20260806 16:15:28.007321 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:27.989550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:50105)
0806 16:15:27.990206 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:27.990272 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:27.991057 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:27.992504 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:27.992512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:27.993015 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:27.993695 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:27.993718 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:27.995550 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:27.995555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:27.996779 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:27.996789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:27.997935 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:27.997939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:27.998135 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:27.998842 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:27.998861 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:28.000940 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:28.003591 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.003598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.003613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.004030 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.004551 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.004554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.004557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.004784 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.005544 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:28.005551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:28.005852 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.006048 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.006391 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":159,"threads_started":1}
I20260806 16:15:28.007339 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:27.989642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:50105 (local address 127.18.90.62:36931)
0806 16:15:27.990094 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:27.990117 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:27.990413 (+ 296us) server_negotiation.cc:407] Connection header received
0806 16:15:27.991542 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:27.991562 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:27.992043 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:27.992432 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:27.993912 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:27.995362 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:27.996913 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:27.997753 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:27.997977 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.001171 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:28.001177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:28.001193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:28.001226 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:28.003449 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.004217 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.004223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.004229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.004431 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.004898 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.004905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.004910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.005411 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:28.005610 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.006219 (+ 609us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.006529 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":131,"threads_started":1}
I20260806 16:15:28.008395 2534 raft_consensus.cc:348] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.008931 2534 raft_consensus.cc:729] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcad3d88247d4050860cfaf6808a8189, State: Initialized, Role: FOLLOWER
I20260806 16:15:28.009442 2534 consensus_queue.cc:260] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.010419 2534 ts_tablet_manager.cc:1428] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:15:28.010531 2644 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36931
I20260806 16:15:28.010759 2644 heartbeater.cc:464] Registering TS with master...
I20260806 16:15:28.011298 2644 heartbeater.cc:511] Master 127.18.90.62:36931 requested a full tablet report, sending...
I20260806 16:15:28.014022 2453 ts_manager.cc:194] Registered new tserver with Master: bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:28.018486 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:50105
I20260806 16:15:28.021498 2644 heartbeater.cc:503] Master 127.18.90.62:36931 was elected leader, sending a full tablet report...
Aug 06 16:15:28 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032927, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.1@KRBTEST.COM
I20260806 16:15:28.028211 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.017353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.0.0.1:44114)
0806 16:15:28.017726 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.017880 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.018013 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.018527 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.018533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.018890 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:28.018995 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.019001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.020235 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.020239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.021202 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.021214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.021925 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.021929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.022052 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.022691 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:28.022705 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:28.024500 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:28.026345 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.026356 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.026361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.026596 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.026920 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.026923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.026925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.027017 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.027488 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:28.027494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:28.027618 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.027757 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.028047 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260806 16:15:28.028211 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.017420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.18.90.1:37061)
0806 16:15:28.018138 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.018150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.018178 (+ 28us) server_negotiation.cc:407] Connection header received
0806 16:15:28.018231 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.018234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.018382 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.018526 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:28.019120 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.020125 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.021355 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.021763 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.021910 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.024641 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:28.024651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:28.024663 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:28.024695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:28.026227 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.026717 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.026723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.026725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.026819 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.027110 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.027115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.027116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.027359 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:28.027495 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.027860 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.028033 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":101,"threads_started":1}
I20260806 16:15:28.029616 2577 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3f528463e4df8bf8d77563d33d406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260806 16:15:28.030514 2653 tablet_replica.cc:330] stopping tablet replica
I20260806 16:15:28.030844 2653 raft_consensus.cc:2229] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:15:28.031103 2653 raft_consensus.cc:2258] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:15:28.032195 2653 ts_tablet_manager.cc:1905] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:15:28.036979 2653 ts_tablet_manager.cc:1918] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:15:28.037092 2653 log.cc:1196] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal/wals/77e3f528463e4df8bf8d77563d33d406
I20260806 16:15:28.037391 2653 ts_tablet_manager.cc:1939] T 77e3f528463e4df8bf8d77563d33d406 P bcad3d88247d4050860cfaf6808a8189: Deleting consensus metadata
I20260806 16:15:28.039618 2436 catalog_manager.cc:4841] TS bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061): tablet 77e3f528463e4df8bf8d77563d33d406 (table test-table [id=385118a08c2148898570956c5884502c]) successfully deleted
W20260806 16:15:28.117889 2648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:28.118137 2648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:28.118206 2648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:28.122279 2648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:28.122393 2648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:28.122457 2648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:15:28.127820 2648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:44317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=37805
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:28.128876 2648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:28.129838 2648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:28.139649 2659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:28.139758 2648 server_base.cc:1034] running on GCE node
W20260806 16:15:28.139649 2660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:28.139693 2662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:28.140445 2648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:28.141201 2648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:28.142602 2648 hybrid_clock.cc:648] HybridClock initialized: now 1786032928142555 us; error 65 us; skew 500 ppm
Aug 06 16:15:28 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032928, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:28.145588 2648 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:15:28.146574 2648 webserver.cc:460] Webserver started at http://127.18.90.2:37805/ using document root <none> and password file <none>
I20260806 16:15:28.147148 2648 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:28.147226 2648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:28.151075 2648 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 16:15:28.153184 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:28.154534 2648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:15:28.154627 2648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "acdf2b6e506a4948933cfc43645936f1"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "35c38ee47f87d17a3f916a539128c890"
server_key_iv: "f1b8f9d0025cb904f12e6f6b89b8f87a"
server_key_version: "encryptionkey@0"
I20260806 16:15:28.155174 2648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:28.171288 2648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:28.174537 2648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:28.174675 2648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:28.175153 2648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:28.175901 2648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:28.175947 2648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:28.175985 2648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:28.176007 2648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:28.185153 2648 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44317
I20260806 16:15:28.185184 2784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44317 every 8 connection(s)
I20260806 16:15:28.186164 2648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:15:28.190785 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2648
I20260806 16:15:28.191277 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:41911
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:36931
--webserver_port=36517
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:15:28 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032928, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:28.201289 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.187973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:53453 (local address 127.18.90.62:36931)
0806 16:15:28.188117 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.188121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.189021 (+ 900us) server_negotiation.cc:407] Connection header received
0806 16:15:28.190319 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.190322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.190372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.190445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:28.191772 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.192636 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.194302 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.194578 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.194697 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.197194 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:28.197199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:28.197206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:28.197245 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:28.199071 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.199583 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.199589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.199593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.199694 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.200033 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.200039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.200043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.200329 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:28.200442 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.200897 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.201055 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:15:28.201907 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.188392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:53453)
0806 16:15:28.188871 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.188914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.190106 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.190575 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.190581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.191030 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:28.191604 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.191620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.192760 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.192763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.194147 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.194160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.194755 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.194759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.194897 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.195485 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:28.195501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:28.197010 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:28.199194 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.199198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.199209 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.199479 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.199809 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.199811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.199812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.199937 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.200431 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:28.200436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:28.200655 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.200808 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.201047 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260806 16:15:28.203090 2785 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36931
I20260806 16:15:28.203393 2785 heartbeater.cc:464] Registering TS with master...
I20260806 16:15:28.204017 2785 heartbeater.cc:511] Master 127.18.90.62:36931 requested a full tablet report, sending...
I20260806 16:15:28.205158 2453 ts_manager.cc:194] Registered new tserver with Master: acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
I20260806 16:15:28.205744 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:53453
W20260806 16:15:28.306444 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:28.306700 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:28.306742 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:28.310638 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:28.310691 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:28.310791 2788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:15:28.315845 2788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=36517
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:28.316852 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:28.317840 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:28.327425 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:28.327489 2788 server_base.cc:1034] running on GCE node
W20260806 16:15:28.327505 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:28.327409 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:28.328059 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:28.328675 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:28.329864 2788 hybrid_clock.cc:648] HybridClock initialized: now 1786032928329860 us; error 35 us; skew 500 ppm
Aug 06 16:15:28 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032928, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:28.332475 2788 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:15:28.333323 2788 webserver.cc:460] Webserver started at http://127.18.90.3:36517/ using document root <none> and password file <none>
I20260806 16:15:28.333868 2788 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:28.333974 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:28.337680 2788 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 16:15:28.339684 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:28.341068 2788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:15:28.341200 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "b961a4432c224396b18946906acec2c8"
format_stamp: "Formatted at 2026-08-06 16:15:07 on dist-test-slave-0b58"
server_key: "4eb92470939d05c2f22ff7b4a140bf24"
server_key_iv: "1caeba0104602c63cca50d4b3992dbe1"
server_key_version: "encryptionkey@0"
I20260806 16:15:28.341786 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:28.370852 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:28.373685 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:28.373911 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:28.374527 2788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:28.375370 2788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:28.375439 2788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:28.375509 2788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:28.375560 2788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:28.384361 2788 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:41911
I20260806 16:15:28.384368 2916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:41911 every 8 connection(s)
I20260806 16:15:28.385392 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:15:28.388859 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2788
Aug 06 16:15:28 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032928, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:15:28.398319 2921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.389557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.18.90.62:36931)
0806 16:15:28.389811 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.389814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.389946 (+ 132us) server_negotiation.cc:407] Connection header received
0806 16:15:28.390077 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.390079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.390117 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.390194 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:28.391131 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.391746 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.392851 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.393062 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.393137 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.394718 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:28.394723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:28.394726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:28.394750 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:28.396483 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.396942 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.396948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.396951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.397055 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.397371 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.397374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.397375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.397622 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:28.397869 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.397977 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.398104 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260806 16:15:28.400471 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.387128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:58309 (local address 127.18.90.62:36931)
0806 16:15:28.387279 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.387283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.388025 (+ 742us) server_negotiation.cc:407] Connection header received
0806 16:15:28.389788 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.389792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.389848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.389909 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:28.391583 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.392122 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.393642 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.393859 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.393968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.396561 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:28.396567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:28.396573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:28.396601 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:28.398120 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.398730 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.398733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.398734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.398841 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.399253 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.399257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.399258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.399487 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:28.399584 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.400165 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.400320 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:15:28.401279 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.387438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:58309)
0806 16:15:28.387876 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.387919 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.389533 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.390374 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.390383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.390884 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:28.391445 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.391458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.392238 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.392242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.393509 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.393520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.394008 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.394012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.394187 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.394735 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:28.394754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:28.396418 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:28.398243 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.398249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.398281 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.398600 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.398942 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:28.398945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:28.398947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:28.399133 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:28.399600 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:28.399607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:28.399887 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.400081 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.400441 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260806 16:15:28.402336 2917 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36931
I20260806 16:15:28.402593 2917 heartbeater.cc:464] Registering TS with master...
I20260806 16:15:28.403076 2917 heartbeater.cc:511] Master 127.18.90.62:36931 requested a full tablet report, sending...
I20260806 16:15:28.404039 2453 ts_manager.cc:194] Registered new tserver with Master: b961a4432c224396b18946906acec2c8 (127.18.90.3:41911)
I20260806 16:15:28.404608 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:58309
I20260806 16:15:28.412210 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:15:28.416003 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.412878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.18.90.62:36931)
0806 16:15:28.413026 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.413029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.413205 (+ 176us) server_negotiation.cc:407] Connection header received
0806 16:15:28.413345 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.413347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.413444 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.413530 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:15:28.413744 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.414232 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.415030 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.415230 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.415310 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.415574 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:15:28.415704 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 16:15:28.415905 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38554: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 16:15:28.416090 2646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38554: token expired
W20260806 16:15:28.416461 2379 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.90.62:36931)
I20260806 16:15:28.423852 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.416833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.18.90.62:36931)
0806 16:15:28.416957 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.416963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.416977 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:15:28.417116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.417119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.417160 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.417250 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:28.417869 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.418344 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.419139 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.419371 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.419490 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.420908 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:28.420910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:28.420912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:28.420951 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:28.422429 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.422796 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.422798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.422800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.422870 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:28.423091 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:28.423094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:28.423095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:28.423256 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:28.423357 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.423561 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.423647 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:15:28.426362 2453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:15:28.429589 2453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:15:28.437387 2577 tablet_service.cc:1467] Processing CreateTablet for tablet ea9d0cd807f147859e792e41720155d5 (DEFAULT_TABLE table=test-table [id=6106d09c4e8e4a1b9fc34e00162dde74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:15:28.438088 2577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9d0cd807f147859e792e41720155d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:28.441020 2534 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Bootstrap starting.
I20260806 16:15:28.441992 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.435932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41911 (local address 127.0.0.1:35408)
0806 16:15:28.436066 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.436085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.436169 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.437453 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.437456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.437475 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.437568 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.437574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.439128 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.439131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.440203 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.440215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.441476 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.441479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.441605 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.441623 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.441708 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.441846 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":33}
I20260806 16:15:28.442021 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.436436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44317 (local address 127.0.0.1:35660)
0806 16:15:28.436658 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.436672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.436759 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.437635 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.437639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.437659 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.437792 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.437798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.438909 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.438915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.440121 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.440131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.441438 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.441441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.441569 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.441617 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.441717 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.441845 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":44,"thread_start_us":103,"threads_started":1}
I20260806 16:15:28.442075 2534 tablet_bootstrap.cc:654] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:28.442171 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.436635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.18.90.2:44317)
0806 16:15:28.437047 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.437058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.437076 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:15:28.437139 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.437144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.437305 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.437448 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.437912 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.438795 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.440244 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.441241 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.441350 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.441532 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.441905 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.442007 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":126,"threads_started":1}
I20260806 16:15:28.443388 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.436359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.18.90.3:41911)
0806 16:15:28.436883 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.436896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.436948 (+ 52us) server_negotiation.cc:407] Connection header received
0806 16:15:28.437034 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.437042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.437278 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.437448 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.437712 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.438990 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.440384 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.441343 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.442889 (+ 1546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.443039 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.443095 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.443194 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":166,"threads_started":1}
I20260806 16:15:28.443652 2534 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: No bootstrap required, opened a new log
I20260806 16:15:28.443727 2534 ts_tablet_manager.cc:1397] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:15:28.444145 2534 raft_consensus.cc:348] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.444245 2534 raft_consensus.cc:374] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:28.444346 2534 raft_consensus.cc:729] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcad3d88247d4050860cfaf6808a8189, State: Initialized, Role: FOLLOWER
I20260806 16:15:28.444485 2534 consensus_queue.cc:260] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.444801 2534 ts_tablet_manager.cc:1428] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:15:28.445340 2717 tablet_service.cc:1467] Processing CreateTablet for tablet ea9d0cd807f147859e792e41720155d5 (DEFAULT_TABLE table=test-table [id=6106d09c4e8e4a1b9fc34e00162dde74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:15:28.446557 2717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9d0cd807f147859e792e41720155d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:28.447854 2851 tablet_service.cc:1467] Processing CreateTablet for tablet ea9d0cd807f147859e792e41720155d5 (DEFAULT_TABLE table=test-table [id=6106d09c4e8e4a1b9fc34e00162dde74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:15:28.448858 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9d0cd807f147859e792e41720155d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:15:28.453807 2930 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Bootstrap starting.
I20260806 16:15:28.456094 2930 tablet_bootstrap.cc:654] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:28.456646 2931 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Bootstrap starting.
I20260806 16:15:28.457376 2930 log.cc:824] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:28.458626 2931 tablet_bootstrap.cc:654] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Neither blocks nor log segments found. Creating new log.
I20260806 16:15:28.459395 2930 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: No bootstrap required, opened a new log
I20260806 16:15:28.459612 2930 ts_tablet_manager.cc:1397] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:15:28.459867 2931 log.cc:824] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:28.461817 2931 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: No bootstrap required, opened a new log
I20260806 16:15:28.462018 2931 ts_tablet_manager.cc:1397] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:15:28.462838 2930 raft_consensus.cc:348] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.463066 2930 raft_consensus.cc:374] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:28.463126 2930 raft_consensus.cc:729] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acdf2b6e506a4948933cfc43645936f1, State: Initialized, Role: FOLLOWER
I20260806 16:15:28.463547 2930 consensus_queue.cc:260] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.464212 2785 heartbeater.cc:503] Master 127.18.90.62:36931 was elected leader, sending a full tablet report...
I20260806 16:15:28.464526 2930 ts_tablet_manager.cc:1428] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:15:28.465354 2931 raft_consensus.cc:348] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.465567 2931 raft_consensus.cc:374] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:15:28.465637 2931 raft_consensus.cc:729] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b961a4432c224396b18946906acec2c8, State: Initialized, Role: FOLLOWER
I20260806 16:15:28.466055 2931 consensus_queue.cc:260] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.466729 2917 heartbeater.cc:503] Master 127.18.90.62:36931 was elected leader, sending a full tablet report...
I20260806 16:15:28.467082 2931 ts_tablet_manager.cc:1428] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 16:15:28.488562 2645 tablet.cc:2369] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:15:28.636494 2918 tablet.cc:2369] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:15:28.687346 2786 tablet.cc:2369] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:15:28.775002 2935 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:28.775238 2935 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.776471 2935 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317), bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:28.781597 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.776765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.18.90.3:45729)
0806 16:15:28.777037 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.777053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.777154 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.777872 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.777878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.777900 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.778010 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.778016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.778815 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.778819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.780315 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.780327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.781165 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.781168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.781264 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.781294 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.781383 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.781458 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":152}
I20260806 16:15:28.781668 2650 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:28.781766 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.776895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:36345 (local address 127.18.90.2:44317)
0806 16:15:28.777044 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.777049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.777378 (+ 329us) server_negotiation.cc:407] Connection header received
0806 16:15:28.777514 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.777523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.777569 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.777667 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.777996 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.778819 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.780445 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.781045 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.781161 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.781228 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.781569 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.781647 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:15:28.781845 2650 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.782104 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.776788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44317 (local address 127.18.90.3:36345)
0806 16:15:28.777260 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.777274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.777394 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.777671 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.777674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.777691 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.777837 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.777848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.778926 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.778929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.780315 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.780327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.781301 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.781304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.781432 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.781457 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.781891 (+ 434us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.781978 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":22,"thread_start_us":120,"threads_started":1}
I20260806 16:15:28.782444 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.777281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:45729 (local address 127.18.90.1:37061)
0806 16:15:28.777546 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.777550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.777563 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:15:28.777616 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.777619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.777684 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.777838 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.778173 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.778673 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.780488 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.780933 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.781170 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.781359 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.781400 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.782319 (+ 919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
I20260806 16:15:28.782949 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcad3d88247d4050860cfaf6808a8189" is_pre_election: true
I20260806 16:15:28.783118 2650 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
I20260806 16:15:28.783167 2598 raft_consensus.cc:2454] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 0.
I20260806 16:15:28.783193 2738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdf2b6e506a4948933cfc43645936f1" is_pre_election: true
I20260806 16:15:28.783501 2738 raft_consensus.cc:2454] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 0.
I20260806 16:15:28.783690 2804 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b961a4432c224396b18946906acec2c8, bcad3d88247d4050860cfaf6808a8189; no voters:
I20260806 16:15:28.783816 2934 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:28.783905 2934 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.783964 2935 raft_consensus.cc:2781] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:15:28.784054 2935 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:15:28.784096 2935 raft_consensus.cc:3037] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:15:28.784978 2934 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:28.785387 2935 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.786069 2738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdf2b6e506a4948933cfc43645936f1"
I20260806 16:15:28.786212 2738 raft_consensus.cc:3037] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:15:28.787488 2738 raft_consensus.cc:2454] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 1.
I20260806 16:15:28.787884 2805 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8; no voters:
I20260806 16:15:28.788673 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "b961a4432c224396b18946906acec2c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcad3d88247d4050860cfaf6808a8189"
I20260806 16:15:28.788802 2598 raft_consensus.cc:3037] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:15:28.789948 2598 raft_consensus.cc:2454] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b961a4432c224396b18946906acec2c8 in term 1.
I20260806 16:15:28.790158 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.783326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41911 (local address 127.18.90.1:60799)
0806 16:15:28.784509 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.784536 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.784622 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.786225 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.786228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.786247 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.786331 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.786336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.787266 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.787269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.788556 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.788564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.789463 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.789466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.789842 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.789865 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.789927 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.789989 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":280,"thread_start_us":79,"threads_started":1}
I20260806 16:15:28.790325 2935 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [CANDIDATE]: Term 1 election: Requested vote from peers acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317), bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061)
I20260806 16:15:28.790591 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.784146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:60799 (local address 127.18.90.3:41911)
0806 16:15:28.785221 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.785226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.785238 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:15:28.785275 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.785279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.785322 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.785394 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.786448 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.787147 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.788692 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.789249 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.789323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.789375 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.790409 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.790485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981}
I20260806 16:15:28.791148 2871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "bcad3d88247d4050860cfaf6808a8189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b961a4432c224396b18946906acec2c8" is_pre_election: true
I20260806 16:15:28.791162 2935 raft_consensus.cc:2781] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:15:28.791478 2935 raft_consensus.cc:686] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 1 LEADER]: Becoming Leader. State: Replica: b961a4432c224396b18946906acec2c8, State: Running, Role: LEADER
I20260806 16:15:28.791747 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.785317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:52375 (local address 127.18.90.2:44317)
0806 16:15:28.785480 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.785484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.786991 (+ 1507us) server_negotiation.cc:407] Connection header received
0806 16:15:28.787139 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.787141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.787207 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.787254 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.787588 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.788320 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.789694 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.790265 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.790336 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.790392 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.791583 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.791657 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 16:15:28.792167 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.783557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44317 (local address 127.18.90.1:52375)
0806 16:15:28.786919 (+ 3362us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.786932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.787045 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.787380 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.787383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.787403 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.787475 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.787480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.788450 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.788453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.789571 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.789578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.791351 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.791355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.791440 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.791461 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.791928 (+ 467us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.792013 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3301,"mutex_wait_us":13,"thread_start_us":70,"threads_started":1}
I20260806 16:15:28.792589 2738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "bcad3d88247d4050860cfaf6808a8189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdf2b6e506a4948933cfc43645936f1" is_pre_election: true
I20260806 16:15:28.792750 2738 raft_consensus.cc:2379] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcad3d88247d4050860cfaf6808a8189 in current term 1: Already voted for candidate b961a4432c224396b18946906acec2c8 in this term.
I20260806 16:15:28.792903 2935 consensus_queue.cc:237] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:28.795094 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.785924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:45311 (local address 127.18.90.3:41911)
0806 16:15:28.786561 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.786569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.786580 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:15:28.786613 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.786616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.786653 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.786727 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.790438 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.791146 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.793206 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.794473 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.794590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.794678 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.794881 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.794952 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":116,"threads_started":1}
I20260806 16:15:28.795590 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.785095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41911 (local address 127.18.90.2:45311)
0806 16:15:28.785679 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.785693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.785775 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.789556 (+ 3781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.789559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.789578 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.789710 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.789716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.791657 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.791761 (+ 104us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.793070 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.793080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.794609 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.794614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.794716 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.794739 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.795369 (+ 630us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.795450 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":86,"threads_started":1}
I20260806 16:15:28.795971 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.785649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.18.90.2:55761)
0806 16:15:28.785995 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.786010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:28.786099 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:28.790848 (+ 4749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:28.790851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:28.790868 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:15:28.790967 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.790989 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.793772 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.793778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.794585 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:28.794595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.795533 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.795536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:28.795636 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.795687 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 16:15:28.795758 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 16:15:28.795824 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
I20260806 16:15:28.796355 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.785694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:55761 (local address 127.18.90.1:37061)
0806 16:15:28.788199 (+ 2505us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.788203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.788215 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:15:28.788251 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.790441 (+ 2190us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.790490 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.790557 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:15:28.791203 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.793655 (+ 2452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.794720 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.795335 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.795432 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.795480 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.796118 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.796187 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2437}
I20260806 16:15:28.796408 2871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "acdf2b6e506a4948933cfc43645936f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b961a4432c224396b18946906acec2c8" is_pre_election: true
I20260806 16:15:28.796706 2530 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcad3d88247d4050860cfaf6808a8189; no voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8
I20260806 16:15:28.797005 2650 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 16:15:28.796988 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9d0cd807f147859e792e41720155d5" candidate_uuid: "acdf2b6e506a4948933cfc43645936f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcad3d88247d4050860cfaf6808a8189" is_pre_election: true
I20260806 16:15:28.797178 2598 raft_consensus.cc:2379] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acdf2b6e506a4948933cfc43645936f1 in current term 1: Already voted for candidate b961a4432c224396b18946906acec2c8 in this term.
I20260806 16:15:28.797253 2453 catalog_manager.cc:5495] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b961a4432c224396b18946906acec2c8 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "b961a4432c224396b18946906acec2c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } health_report { overall_health: UNKNOWN } } }
I20260806 16:15:28.797595 2670 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acdf2b6e506a4948933cfc43645936f1; no voters: b961a4432c224396b18946906acec2c8, bcad3d88247d4050860cfaf6808a8189
I20260806 16:15:28.797848 2934 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 16:15:28.807380 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.803866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.18.90.3:41911)
0806 16:15:28.803988 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:28.803992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:28.804033 (+ 41us) server_negotiation.cc:407] Connection header received
0806 16:15:28.804239 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:28.804242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:28.804312 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:28.804420 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:15:28.804656 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.805138 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.805980 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:28.806229 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:28.806342 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:28.806644 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:15:28.807016 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:15:28.807146 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:28.807203 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 16:15:28.807251 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:15:28.813125 2738 raft_consensus.cc:1261] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Refusing update from remote peer b961a4432c224396b18946906acec2c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:15:28.813123 2598 raft_consensus.cc:1261] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Refusing update from remote peer b961a4432c224396b18946906acec2c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:15:28.813709 2943 consensus_queue.cc:1035] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:15:28.813899 2935 consensus_queue.cc:1035] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:15:28.823173 2948 mvcc.cc:205] Tried to move back new op lower bound from 7315590876410761216 to 7315590876339458048. Current Snapshot: MvccSnapshot[applied={T|T < 7315590876410761216}]
I20260806 16:15:28.827528 2453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:15:28.827672 2453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:15:28.829869 2453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Sending DeleteTablet for 3 replicas of tablet ea9d0cd807f147859e792e41720155d5
I20260806 16:15:28.830524 2851 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9d0cd807f147859e792e41720155d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:28 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:35408
I20260806 16:15:28.830606 2717 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9d0cd807f147859e792e41720155d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:28 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:35660
I20260806 16:15:28.830971 2577 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9d0cd807f147859e792e41720155d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:15:28 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260806 16:15:28.831039 2954 tablet_replica.cc:330] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: stopping tablet replica
I20260806 16:15:28.831347 2956 tablet_replica.cc:330] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: stopping tablet replica
I20260806 16:15:28.831378 2954 raft_consensus.cc:2229] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:15:28.831537 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 2491
I20260806 16:15:28.831528 2956 raft_consensus.cc:2229] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:15:28.831676 2956 raft_consensus.cc:2258] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:15:28.831784 2954 raft_consensus.cc:2258] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:15:28.832396 2955 tablet_replica.cc:330] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: stopping tablet replica
I20260806 16:15:28.832670 2955 raft_consensus.cc:2229] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:15:28.832984 2955 raft_consensus.cc:2258] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:15:28.834585 2954 ts_tablet_manager.cc:1905] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:15:28.834625 2955 ts_tablet_manager.cc:1905] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:15:28.837913 2954 ts_tablet_manager.cc:1918] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:15:28.838085 2954 log.cc:1196] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/ea9d0cd807f147859e792e41720155d5
I20260806 16:15:28.838248 2955 ts_tablet_manager.cc:1918] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:15:28.838400 2955 log.cc:1196] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/ea9d0cd807f147859e792e41720155d5
I20260806 16:15:28.838593 2954 ts_tablet_manager.cc:1939] T ea9d0cd807f147859e792e41720155d5 P acdf2b6e506a4948933cfc43645936f1: Deleting consensus metadata
I20260806 16:15:28.838788 2955 ts_tablet_manager.cc:1939] T ea9d0cd807f147859e792e41720155d5 P b961a4432c224396b18946906acec2c8: Deleting consensus metadata
I20260806 16:15:28.839974 2437 catalog_manager.cc:4841] TS acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): tablet ea9d0cd807f147859e792e41720155d5 (table test-table [id=6106d09c4e8e4a1b9fc34e00162dde74]) successfully deleted
I20260806 16:15:28.840026 2438 catalog_manager.cc:4841] TS b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): tablet ea9d0cd807f147859e792e41720155d5 (table test-table [id=6106d09c4e8e4a1b9fc34e00162dde74]) successfully deleted
W20260806 16:15:28.843089 2436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:37061 (error 108)
I20260806 16:15:28.843402 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 2648
I20260806 16:15:28.844779 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:28.844414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:37061 (local address 127.0.0.1:44122)
0806 16:15:28.844594 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:28.844675 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 16:15:28.844920 2436 catalog_manager.cc:4584] TS bcad3d88247d4050860cfaf6808a8189 (127.18.90.1:37061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea9d0cd807f147859e792e41720155d5: Network error: Client connection negotiation failed: client connection to 127.18.90.1:37061: connect: Connection refused (error 111)
I20260806 16:15:28.851617 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 2788
I20260806 16:15:28.858206 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 2419
I20260806 16:15:48.865784 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36931
--webserver_interface=127.18.90.62
--webserver_port=34683
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:48.979413 2958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:48.979708 2958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:48.979791 2958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:48.983610 2958 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 16:15:48.983692 2958 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 16:15:48.983719 2958 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 16:15:48.983750 2958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:15:48.983791 2958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:48.983829 2958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:48.983865 2958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:15:48.983898 2958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:15:48.989061 2958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=34683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:48.990118 2958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:48.991119 2958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:15:48.999728 2958 server_base.cc:1034] running on GCE node
W20260806 16:15:48.999624 2964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:48.999634 2963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:48.999624 2966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:49.000615 2958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:49.001658 2958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:49.002853 2958 hybrid_clock.cc:648] HybridClock initialized: now 1786032949002844 us; error 30 us; skew 500 ppm
Aug 06 16:15:49 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032949, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:49.011449 2958 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:15:49.012483 2958 webserver.cc:460] Webserver started at http://127.18.90.62:34683/ using document root <none> and password file <none>
I20260806 16:15:49.013065 2958 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:49.013129 2958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:49.017055 2958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:15:49.019148 2973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.020450 2958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:15:49.020586 2958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "5d2f777eda55d0a409a43e659a718d67"
server_key_iv: "e87d12db650776ebf3a20c1f0b48969f"
server_key_version: "encryptionkey@0"
I20260806 16:15:49.021142 2958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:49.032081 2958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:49.035013 2958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:49.035224 2958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:49.042824 2958 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:36931
I20260806 16:15:49.043146 3025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:36931 every 8 connection(s)
I20260806 16:15:49.043895 2958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:15:49.048098 3026 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:15:49.049754 3026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap starting.
I20260806 16:15:49.053260 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 2958
I20260806 16:15:49.053789 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:37061
--local_ip_for_outbound_sockets=127.18.90.1
--tserver_master_addrs=127.18.90.62:36931
--webserver_port=34257
--webserver_interface=127.18.90.1
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:15:49.068491 3026 log.cc:824] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:49.079977 3026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:15:49.080425 3026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Bootstrap complete.
I20260806 16:15:49.084367 3026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:49.084743 3026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc362ab9ae74d27ac37c5b5a1d5b1aa, State: Initialized, Role: FOLLOWER
I20260806 16:15:49.085284 3026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:49.085404 3026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:15:49.085493 3026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:15:49.085605 3026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 2 FOLLOWER]: Advancing to term 3
I20260806 16:15:49.087450 3026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:49.087767 3026 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc362ab9ae74d27ac37c5b5a1d5b1aa; no voters:
I20260806 16:15:49.088076 3026 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 16:15:49.088212 3030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 3 FOLLOWER]: Leader election won for term 3
I20260806 16:15:49.088495 3030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [term 3 LEADER]: Becoming Leader. State: Replica: 7bc362ab9ae74d27ac37c5b5a1d5b1aa, State: Running, Role: LEADER
I20260806 16:15:49.088904 3030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } }
I20260806 16:15:49.089299 3026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:15:49.090407 3032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc362ab9ae74d27ac37c5b5a1d5b1aa. Latest consensus state: current_term: 3 leader_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } } }
I20260806 16:15:49.090528 3032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: This master's current role is: LEADER
I20260806 16:15:49.090879 3039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:15:49.090891 3031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc362ab9ae74d27ac37c5b5a1d5b1aa" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36931 } } }
I20260806 16:15:49.091013 3031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa [sys.catalog]: This master's current role is: LEADER
I20260806 16:15:49.095624 3039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:15:49.096446 3039 catalog_manager.cc:1259] Loaded cluster ID: 6a20c46b81f441eabdc36d82ad1dca51
I20260806 16:15:49.096522 3039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:15:49.098264 3039 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:15:49.098943 3039 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Loaded TSK: 0
I20260806 16:15:49.099025 3039 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7bc362ab9ae74d27ac37c5b5a1d5b1aa: Loaded TSK: 1
F20260806 16:15:49.099707 3039 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcf2741c9ad google::LogMessage::Fail() at ??:0
@ 0x7fcf2741f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcf2741c4de google::LogMessage::Flush() at ??:0
@ 0x7fcf2741da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcf2a0e018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcf2a0d8f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcf31d857bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcf31d5f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcf31d60178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fcf31da1df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fcf31dc082b std::function<>::operator()() at ??:0
@ 0x7fcf31d5fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fcf31d60f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcf31d5d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fcf31da1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcf2966e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcf2966ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcf296703f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcf2965f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcf2aca36db start_thread at ??:0
@ 0x7fcf25f1a61f clone at ??:0
W20260806 16:15:49.208904 3028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:49.209161 3028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:49.209206 3028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:49.213318 3028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:49.213373 3028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:49.213476 3028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:15:49.218868 3028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:37061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=34257
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:49.219985 3028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:49.221086 3028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:49.231244 3073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:49.231415 3028 server_base.cc:1034] running on GCE node
W20260806 16:15:49.231294 3074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:49.231664 3077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:49.232053 3028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:49.232755 3028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:49.233944 3028 hybrid_clock.cc:648] HybridClock initialized: now 1786032949233918 us; error 61 us; skew 500 ppm
Aug 06 16:15:49 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032949, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:49.236686 3028 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:15:49.237659 3028 webserver.cc:460] Webserver started at http://127.18.90.1:34257/ using document root <none> and password file <none>
I20260806 16:15:49.238220 3028 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:49.238286 3028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:49.242445 3028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 16:15:49.244633 3084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.245939 3028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:15:49.246042 3028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "bcad3d88247d4050860cfaf6808a8189"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "64fd017ab4f36eefef4d4350b8c0cd2d"
server_key_iv: "bc01279c0e02cb1075aa879e41631c6c"
server_key_version: "encryptionkey@0"
I20260806 16:15:49.246613 3028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:49.280496 3028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:49.284034 3028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:49.284282 3028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:49.284991 3028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:49.286163 3091 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:15:49.290331 3028 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:15:49.290397 3028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260806 16:15:49.290467 3028 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:15:49.292236 3028 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:15:49.292337 3028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 16:15:49.292606 3091 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Bootstrap starting.
I20260806 16:15:49.303388 3028 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:37061
I20260806 16:15:49.303524 3200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:37061 every 8 connection(s)
I20260806 16:15:49.304878 3028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:15:49.312290 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3028
I20260806 16:15:49.312860 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:44317
--local_ip_for_outbound_sockets=127.18.90.2
--tserver_master_addrs=127.18.90.62:36931
--webserver_port=37805
--webserver_interface=127.18.90.2
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:15:49.313809 3091 log.cc:824] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Log is configured to *not* fsync() on all Append() calls
I20260806 16:15:49.321764 3091 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:15:49.322289 3091 tablet_bootstrap.cc:492] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Bootstrap complete.
I20260806 16:15:49.324186 3091 ts_tablet_manager.cc:1397] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20260806 16:15:49.328635 3091 raft_consensus.cc:348] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:49.329377 3091 raft_consensus.cc:729] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcad3d88247d4050860cfaf6808a8189, State: Initialized, Role: FOLLOWER
W20260806 16:15:49.330123 3203 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:15:49.307312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:60103)
0806 16:15:49.307926 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.307988 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:49.309153 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:49.329096 (+ 19943us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":170,"threads_started":1}
I20260806 16:15:49.330066 3091 consensus_queue.cc:260] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
W20260806 16:15:49.330572 3201 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:36931 (0 consecutive failures): Network error: Failed to ping master at 127.18.90.62:36931: Client connection negotiation failed: client connection to 127.18.90.62:36931: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:15:49.331287 3091 ts_tablet_manager.cc:1428] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 16:15:49.331460 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.331014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:48409)
0806 16:15:49.331313 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.331396 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260806 16:15:49.332195 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.331986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:52111)
0806 16:15:49.332097 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.332148 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 16:15:49.332410 3201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:15:49.432193 3205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:49.432502 3205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:49.432577 3205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:49.436546 3205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:49.436597 3205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:49.436652 3205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:15:49.441910 3205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:44317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=37805
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:49.442917 3205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:49.443943 3205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:49.453773 3212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:49.453847 3215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:49.453773 3213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:49.453974 3205 server_base.cc:1034] running on GCE node
I20260806 16:15:49.454483 3205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:49.455118 3205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:49.456337 3205 hybrid_clock.cc:648] HybridClock initialized: now 1786032949456323 us; error 25 us; skew 500 ppm
Aug 06 16:15:49 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032949, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:49.459044 3205 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:15:49.459929 3205 webserver.cc:460] Webserver started at http://127.18.90.2:37805/ using document root <none> and password file <none>
I20260806 16:15:49.460529 3205 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:49.460593 3205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:49.464437 3205 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 16:15:49.466450 3222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.467600 3205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:15:49.467723 3205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "acdf2b6e506a4948933cfc43645936f1"
format_stamp: "Formatted at 2026-08-06 16:15:06 on dist-test-slave-0b58"
server_key: "35c38ee47f87d17a3f916a539128c890"
server_key_iv: "f1b8f9d0025cb904f12e6f6b89b8f87a"
server_key_version: "encryptionkey@0"
I20260806 16:15:49.468238 3205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:49.481940 3205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:49.485008 3205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:49.485230 3205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:49.485803 3205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:49.486625 3205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:49.486728 3205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.486824 3205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:49.486876 3205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.495539 3205 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44317
I20260806 16:15:49.495573 3337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44317 every 8 connection(s)
I20260806 16:15:49.496567 3205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:15:49.500188 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.498564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:53349)
0806 16:15:49.498996 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.499096 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
W20260806 16:15:49.500770 3338 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:36931 (0 consecutive failures): Network error: Failed to ping master at 127.18.90.62:36931: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
I20260806 16:15:49.501318 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.501119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:43509)
0806 16:15:49.501218 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.501265 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 16:15:49.501874 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.501698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:33345)
0806 16:15:49.501792 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.501830 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 16:15:49.502058 3338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:15:49.505138 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3205
I20260806 16:15:49.505555 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:41911
--local_ip_for_outbound_sockets=127.18.90.3
--tserver_master_addrs=127.18.90.62:36931
--webserver_port=36517
--webserver_interface=127.18.90.3
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:15:49.618139 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:15:49.618431 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:15:49.618476 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:15:49.622593 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:15:49.622656 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:15:49.622714 3341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:15:49.628064 3341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=36517
--tserver_master_addrs=127.18.90.62:36931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:15:49.629222 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:15:49.630213 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:15:49.639994 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:49.640007 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:15:49.640087 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:15:49.640491 3341 server_base.cc:1034] running on GCE node
I20260806 16:15:49.640990 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:15:49.641709 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:15:49.642904 3341 hybrid_clock.cc:648] HybridClock initialized: now 1786032949642875 us; error 52 us; skew 500 ppm
Aug 06 16:15:49 dist-test-slave-0b58 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032949, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:15:49.645694 3341 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:15:49.646621 3341 webserver.cc:460] Webserver started at http://127.18.90.3:36517/ using document root <none> and password file <none>
I20260806 16:15:49.647195 3341 fs_manager.cc:359] Metadata directory not provided
I20260806 16:15:49.647296 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:15:49.651539 3341 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:15:49.653797 3357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.655059 3341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:15:49.655242 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "b961a4432c224396b18946906acec2c8"
format_stamp: "Formatted at 2026-08-06 16:15:07 on dist-test-slave-0b58"
server_key: "4eb92470939d05c2f22ff7b4a140bf24"
server_key_iv: "1caeba0104602c63cca50d4b3992dbe1"
server_key_version: "encryptionkey@0"
I20260806 16:15:49.655843 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:15:49.689011 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:15:49.692054 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:15:49.692308 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:15:49.692953 3341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:15:49.693835 3341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:15:49.693905 3341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.693979 3341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:15:49.694032 3341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:15:49.703194 3341 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:41911
I20260806 16:15:49.703223 3471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:41911 every 8 connection(s)
I20260806 16:15:49.704238 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:15:49.708351 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.706548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:52929)
0806 16:15:49.707472 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.707588 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":573,"threads_started":1}
W20260806 16:15:49.708858 3472 heartbeater.cc:644] Failed to heartbeat to 127.18.90.62:36931 (0 consecutive failures): Network error: Failed to ping master at 127.18.90.62:36931: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
I20260806 16:15:49.709491 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.709283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:58705)
0806 16:15:49.709388 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.709436 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 16:15:49.710052 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:49.709878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:53731)
0806 16:15:49.709968 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:49.710007 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 16:15:49.710238 3472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:15:49.713155 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3341
I20260806 16:15:50.333415 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.332969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:35051)
0806 16:15:50.333245 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:50.333300 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260806 16:15:50.503093 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.502666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:56815)
0806 16:15:50.502928 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:50.502989 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260806 16:15:50.597105 3478 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:50.597373 3478 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:50.598570 3478 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
Aug 06 16:15:50 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032949, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.2@KRBTEST.COM
Aug 06 16:15:50 dist-test-slave-0b58 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032949, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.3@KRBTEST.COM
I20260806 16:15:50.612300 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.599001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:59347 (local address 127.18.90.2:44317)
0806 16:15:50.599464 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:50.599484 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:50.599517 (+ 33us) server_negotiation.cc:407] Connection header received
0806 16:15:50.600157 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:50.600162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:50.600592 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:50.600896 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:50.602164 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.603234 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.604518 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.604982 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.605172 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:50.607605 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:50.607616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:50.607634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:50.607671 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:50.609458 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610053 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:50.610078 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:50.610085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:50.610239 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610615 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:50.610618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:50.610619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:50.611114 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:50.611219 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:50.611830 (+ 611us) server_negotiation.cc:299] Negotiation successful
0806 16:15:50.612079 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":94,"threads_started":1}
I20260806 16:15:50.612290 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.598781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41911 (local address 127.18.90.1:55253)
0806 16:15:50.598975 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:50.598991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:50.599098 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:50.600952 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:50.600958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:50.601467 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:50.601948 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:50.601965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.603379 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.603383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:50.604392 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:50.604402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.605219 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.605222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:50.605307 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:50.606021 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:50.606034 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:50.607706 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:50.610113 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:50.610118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:50.610400 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:50.610757 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:50.610762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:50.610850 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:50.611465 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:50.611467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:50.611553 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 16:15:50.611706 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 16:15:50.612052 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":103,"spinlock_wait_cycles":33408}
I20260806 16:15:50.612322 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.598993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:55253 (local address 127.18.90.3:41911)
0806 16:15:50.599378 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 16:15:50.599391 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:15:50.599428 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:15:50.600106 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:15:50.600112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:15:50.600592 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:15:50.600897 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:15:50.602136 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.603233 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.604514 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.604990 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.605277 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:50.607893 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:15:50.607897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:15:50.607908 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:15:50.607937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:15:50.610011 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610504 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:50.610510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:50.610514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:50.610656 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610940 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:15:50.610942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:15:50.610943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:15:50.611358 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:15:50.611485 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:15:50.611822 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 16:15:50.612109 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":84,"threads_started":1}
I20260806 16:15:50.612346 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.598853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44317 (local address 127.18.90.1:59347)
0806 16:15:50.599200 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:50.599215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:15:50.599315 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:15:50.600952 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:15:50.600958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:15:50.601493 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:15:50.601947 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:50.601968 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.603383 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.603386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:50.604397 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:15:50.604406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:15:50.605114 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:15:50.605118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:15:50.605283 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:15:50.605857 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:15:50.605873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:15:50.607354 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:15:50.609581 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:50.609587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:50.609602 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:50.609941 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:50.610340 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:15:50.610342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:15:50.610344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:15:50.610515 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:15:50.611233 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:15:50.611239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:15:50.611490 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 16:15:50.611707 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 16:15:50.612136 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":250,"spinlock_wait_cycles":19456,"thread_start_us":76,"threads_started":1}
W20260806 16:15:50.614984 3086 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
W20260806 16:15:50.615211 3087 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
I20260806 16:15:50.615315 3087 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcad3d88247d4050860cfaf6808a8189; no voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8
I20260806 16:15:50.615583 3478 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:15:50.711293 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:50.710819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:34053)
0806 16:15:50.711088 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:50.711172 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260806 16:15:51.334679 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:51.334225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:60795)
0806 16:15:51.334517 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:51.334576 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260806 16:15:51.504400 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:51.503854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:48595)
0806 16:15:51.504156 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:51.504228 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260806 16:15:51.712620 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:51.712074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:33013)
0806 16:15:51.712464 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:51.712516 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":239,"threads_started":1}
I20260806 16:15:52.335860 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:52.335449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:59761)
0806 16:15:52.335708 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:52.335762 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260806 16:15:52.505592 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:52.505158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:52987)
0806 16:15:52.505436 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:52.505497 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260806 16:15:52.713784 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:52.713360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:60077)
0806 16:15:52.713627 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:52.713687 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260806 16:15:52.731343 3490 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:52.731503 3490 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:52.731990 3490 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
W20260806 16:15:52.732867 3087 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
W20260806 16:15:52.732961 3086 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
I20260806 16:15:52.733088 3086 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcad3d88247d4050860cfaf6808a8189; no voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8
I20260806 16:15:52.733263 3490 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:15:53.337320 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:53.336688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:59793)
0806 16:15:53.337125 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:53.337194 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260806 16:15:53.506814 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:53.506373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:51003)
0806 16:15:53.506657 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:53.506718 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260806 16:15:53.714957 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:53.714545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:34771)
0806 16:15:53.714804 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:53.714865 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260806 16:15:54.338824 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:54.338340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:52979)
0806 16:15:54.338650 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:54.338713 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
I20260806 16:15:54.508005 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:54.507576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:56857)
0806 16:15:54.507850 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:54.507907 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260806 16:15:54.716101 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:54.715684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:52521)
0806 16:15:54.715946 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:54.716004 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260806 16:15:55.118361 3497 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:55.118523 3497 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:55.119004 3497 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
W20260806 16:15:55.119798 3087 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
W20260806 16:15:55.119931 3086 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
I20260806 16:15:55.120048 3086 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcad3d88247d4050860cfaf6808a8189; no voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8
I20260806 16:15:55.120239 3497 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:15:55.340108 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:55.339576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:51051)
0806 16:15:55.339951 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:55.340010 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
I20260806 16:15:55.509358 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:55.508808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:33323)
0806 16:15:55.509170 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:55.509241 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260806 16:15:55.717193 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:55.716805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:36211)
0806 16:15:55.717048 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:55.717102 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260806 16:15:56.341274 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:56.340877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:48959)
0806 16:15:56.341119 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:56.341176 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260806 16:15:56.510627 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:56.510139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:53159)
0806 16:15:56.510466 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:56.510527 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
I20260806 16:15:56.718349 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:56.717899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:60155)
0806 16:15:56.718198 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:56.718252 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260806 16:15:57.342458 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:57.342068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:40677)
0806 16:15:57.342316 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:57.342366 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260806 16:15:57.511804 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:57.511394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:46981)
0806 16:15:57.511643 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:57.511699 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260806 16:15:57.719439 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:57.719046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:46383)
0806 16:15:57.719289 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:57.719342 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260806 16:15:58.343717 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:58.343300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:38025)
0806 16:15:58.343556 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:58.343616 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260806 16:15:58.513181 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:58.512687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:45179)
0806 16:15:58.513024 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:58.513078 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":129,"threads_started":1}
I20260806 16:15:58.720948 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:58.720400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:45943)
0806 16:15:58.720755 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:58.720837 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":80,"threads_started":1}
I20260806 16:15:58.820722 3510 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:15:58.820880 3510 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:15:58.821292 3510 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
W20260806 16:15:58.822103 3086 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
W20260806 16:15:58.822225 3087 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
I20260806 16:15:58.822343 3087 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcad3d88247d4050860cfaf6808a8189; no voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8
I20260806 16:15:58.822504 3510 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:15:59.344998 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:59.344599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:43291)
0806 16:15:59.344850 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:59.344900 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260806 16:15:59.514374 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:59.513946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:57481)
0806 16:15:59.514219 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:59.514276 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260806 16:15:59.722287 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:15:59.721835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:51571)
0806 16:15:59.722112 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:15:59.722179 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260806 16:16:00.346437 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:00.345866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:37527)
0806 16:16:00.346208 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:00.346295 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260806 16:16:00.515550 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:00.515155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:37311)
0806 16:16:00.515397 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:00.515451 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260806 16:16:00.723543 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:00.723034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:40965)
0806 16:16:00.723387 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:00.723445 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":75,"threads_started":1}
I20260806 16:16:01.347606 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:01.347180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:47733)
0806 16:16:01.347460 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:01.347511 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260806 16:16:01.516963 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:01.516541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:49465)
0806 16:16:01.516814 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:01.516868 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260806 16:16:01.724946 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:01.724496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:44441)
0806 16:16:01.724785 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:01.724848 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260806 16:16:02.348793 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:02.348378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:51705)
0806 16:16:02.348631 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:02.348687 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260806 16:16:02.518360 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:02.517819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:43377)
0806 16:16:02.518192 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:02.518258 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":98,"threads_started":1}
I20260806 16:16:02.726105 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:02.725686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:36025)
0806 16:16:02.725946 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:02.726006 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260806 16:16:03.350005 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:03.349551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:39879)
0806 16:16:03.349848 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:03.349907 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260806 16:16:03.519716 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:03.519243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:60317)
0806 16:16:03.519542 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:03.519603 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
I20260806 16:16:03.727434 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:03.726965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:35981)
0806 16:16:03.727268 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:03.727333 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260806 16:16:04.351527 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:04.351107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.1:49751)
0806 16:16:04.351370 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:04.351430 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260806 16:16:04.521045 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:04.520614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.2:36141)
0806 16:16:04.520890 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:04.520947 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260806 16:16:04.595655 3528 raft_consensus.cc:482] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:04.595829 3528 raft_consensus.cc:504] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b961a4432c224396b18946906acec2c8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41911 } } peers { permanent_uuid: "acdf2b6e506a4948933cfc43645936f1" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44317 } } peers { permanent_uuid: "bcad3d88247d4050860cfaf6808a8189" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 37061 } }
I20260806 16:16:04.596312 3528 leader_election.cc:288] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b961a4432c224396b18946906acec2c8 (127.18.90.3:41911), acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317)
W20260806 16:16:04.597127 3086 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer acdf2b6e506a4948933cfc43645936f1 (127.18.90.2:44317): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
W20260806 16:16:04.597242 3087 leader_election.cc:341] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b961a4432c224396b18946906acec2c8 (127.18.90.3:41911): Not found: Tablet not found: ea9d0cd807f147859e792e41720155d5
I20260806 16:16:04.597357 3087 leader_election.cc:302] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcad3d88247d4050860cfaf6808a8189; no voters: acdf2b6e506a4948933cfc43645936f1, b961a4432c224396b18946906acec2c8
I20260806 16:16:04.597520 3528 raft_consensus.cc:2726] T ea9d0cd807f147859e792e41720155d5 P bcad3d88247d4050860cfaf6808a8189 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:16:04.715317 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3028
I20260806 16:16:04.722961 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3205
I20260806 16:16:04.728830 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:04.728409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36931 (local address 127.18.90.3:34647)
0806 16:16:04.728670 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:04.728726 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.62:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260806 16:16:04.729346 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3341
2026-08-06T16:16:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59905 ms)
[----------] 30 tests from SecurityITest (243724 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:04 dist-test-slave-0b58 krb5kdc[3532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:04 dist-test-slave-0b58 krb5kdc[3532](info): set up 2 sockets
Aug 06 16:16:04 dist-test-slave-0b58 krb5kdc[3532](info): commencing operation
krb5kdc: starting...
W20260806 16:16:05.768962 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:05 dist-test-slave-0b58 krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:05.895561 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:40075
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:06.009547 3545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:06.009882 3545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:06.009968 3545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:06.014097 3545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:06.014206 3545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:06.014253 3545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:06.014292 3545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:06.014323 3545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:06.019798 3545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:06.021025 3545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:06.022125 3545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:06.030843 3553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.030901 3545 server_base.cc:1034] running on GCE node
W20260806 16:16:06.030843 3550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:06.030844 3551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.031738 3545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:06.033015 3545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:06.034209 3545 hybrid_clock.cc:648] HybridClock initialized: now 1786032966034192 us; error 32 us; skew 500 ppm
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:06.037127 3545 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:06.038041 3545 webserver.cc:460] Webserver started at http://127.18.90.62:44751/ using document root <none> and password file <none>
I20260806 16:16:06.038625 3545 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:06.038719 3545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:06.038971 3545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:06.041204 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "57be56f43c175414c1c6a1b8bdd4cf8d"
server_key_iv: "e5ca72ec91bc2ea1730736cfaffe724e"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.041824 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "57be56f43c175414c1c6a1b8bdd4cf8d"
server_key_iv: "e5ca72ec91bc2ea1730736cfaffe724e"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.045689 3545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:16:06.048136 3560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.049438 3545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:16:06.049587 3545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "57be56f43c175414c1c6a1b8bdd4cf8d"
server_key_iv: "e5ca72ec91bc2ea1730736cfaffe724e"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.049708 3545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:06.066242 3545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:06.069841 3545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:06.070070 3545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:06.077585 3545 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:40075
I20260806 16:16:06.077608 3613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:40075 every 8 connection(s)
I20260806 16:16:06.078598 3545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:06.081403 3614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:06.082453 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3545
I20260806 16:16:06.082592 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:06.082934 18792 external_mini_cluster.cc:1428] Setting key 7d947cde163d7e3eebec8b9297fee5a7
I20260806 16:16:06.087663 3614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a: Bootstrap starting.
I20260806 16:16:06.090029 3614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:06.090924 3614 log.cc:824] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:06.093559 3614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a: No bootstrap required, opened a new log
I20260806 16:16:06.097211 3614 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40075 } }
I20260806 16:16:06.097505 3614 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:06.097590 3614 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36c4d64681f40a0b9dcbf22c1d0bb3a, State: Initialized, Role: FOLLOWER
I20260806 16:16:06.098039 3614 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40075 } }
I20260806 16:16:06.098203 3614 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:06.098289 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.084520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.18.90.62:40075)
0806 16:16:06.084972 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.084987 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.085021 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:16:06.085642 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.085659 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.086007 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.086372 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:06.087235 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.088337 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.089232 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.089719 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.089907 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.092833 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:06.092840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:06.092856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:06.092889 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:06.095085 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.095554 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.095559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.095563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.095676 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.095967 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.095971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.095973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.096457 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:06.096671 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.097005 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.097316 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260806 16:16:06.098294 3614 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:06.098465 3614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:06.099386 3614 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40075 } }
I20260806 16:16:06.099740 3614 leader_election.cc:302] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36c4d64681f40a0b9dcbf22c1d0bb3a; no voters:
I20260806 16:16:06.100073 3614 leader_election.cc:288] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:06.100175 3620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:06.100412 3620 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [term 1 LEADER]: Becoming Leader. State: Replica: d36c4d64681f40a0b9dcbf22c1d0bb3a, State: Running, Role: LEADER
I20260806 16:16:06.100726 3620 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40075 } }
I20260806 16:16:06.101435 3614 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:06.101852 3620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36c4d64681f40a0b9dcbf22c1d0bb3a. Latest consensus state: current_term: 1 leader_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40075 } } }
I20260806 16:16:06.101992 3620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:06.101850 3622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36c4d64681f40a0b9dcbf22c1d0bb3a" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40075 } } }
I20260806 16:16:06.102346 3624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:06.102360 3622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:06.105505 3624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:06.111085 3624 catalog_manager.cc:1347] Generated new cluster ID: 2b85b67b575e434bbd0376d84f99243a
I20260806 16:16:06.111163 3624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:06.128803 3624 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:06.130168 3624 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:06.138192 3624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a: Generated new TSK 0
I20260806 16:16:06.138952 3624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:16:06.176091 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:40075
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:06.296468 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:06.296748 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:06.296792 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:06.300926 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:06.300992 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:06.301081 3641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:16:06.306831 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:06.308024 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:06.309221 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:06.319219 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:06.319221 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.319303 3641 server_base.cc:1034] running on GCE node
W20260806 16:16:06.319315 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.319878 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:06.320602 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:06.321811 3641 hybrid_clock.cc:648] HybridClock initialized: now 1786032966321778 us; error 43 us; skew 500 ppm
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:06.324754 3641 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:16:06.325718 3641 webserver.cc:460] Webserver started at http://127.18.90.1:37277/ using document root <none> and password file <none>
I20260806 16:16:06.326318 3641 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:06.326423 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:06.326664 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:06.328928 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "d5f58e711e6a40e08b33997db3e5b542"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "77a5401adcf9be98075213d06a71c7c4"
server_key_iv: "c5322028ba44ecb10509fe03daa894b7"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.329532 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f58e711e6a40e08b33997db3e5b542"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "77a5401adcf9be98075213d06a71c7c4"
server_key_iv: "c5322028ba44ecb10509fe03daa894b7"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.333379 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:16:06.335851 3656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.337153 3641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:06.337296 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "d5f58e711e6a40e08b33997db3e5b542"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "77a5401adcf9be98075213d06a71c7c4"
server_key_iv: "c5322028ba44ecb10509fe03daa894b7"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.337427 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:06.356410 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:06.359913 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:06.360138 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:06.360810 3641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:06.361678 3641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:06.361752 3641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.361820 3641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:06.361871 3641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.371107 3641 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:35441
I20260806 16:16:06.371133 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:35441 every 8 connection(s)
I20260806 16:16:06.372098 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:16:06.373966 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3641
I20260806 16:16:06.374099 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:16:06.374399 18792 external_mini_cluster.cc:1428] Setting key 5d8f6a30f6d394b22d7839fa405bedee
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:16:06.388206 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.374021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:59743 (local address 127.18.90.62:40075)
0806 16:16:06.374183 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.374188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.375140 (+ 952us) server_negotiation.cc:407] Connection header received
0806 16:16:06.376673 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.376679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.376746 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.376839 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:06.378202 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.378734 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.380589 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.380849 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.380962 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.384230 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:06.384233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:06.384235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:06.384329 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:06.385877 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.386487 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.386491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.386493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.386583 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.386946 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.386950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.386952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.387155 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:06.387270 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.387867 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.388027 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:16:06.389482 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.374491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:40075 (local address 127.18.90.1:59743)
0806 16:16:06.374984 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.375043 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.376441 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.377003 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.377010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.377366 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:06.378029 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.378051 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.378867 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.378872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.380459 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.380476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.381043 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.381050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.381290 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.382015 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:06.382042 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:06.384025 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:06.385994 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:06.386000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:06.386014 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:06.386298 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:06.386686 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:06.386690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:06.386692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:06.386841 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:06.387269 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:06.387277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:06.387615 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.387855 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.388229 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260806 16:16:06.390828 3770 heartbeater.cc:347] Connected to a master server at 127.18.90.62:40075
I20260806 16:16:06.391115 3770 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:06.391778 3770 heartbeater.cc:511] Master 127.18.90.62:40075 requested a full tablet report, sending...
I20260806 16:16:06.393658 3577 ts_manager.cc:194] Registered new tserver with Master: d5f58e711e6a40e08b33997db3e5b542 (127.18.90.1:35441)
I20260806 16:16:06.395424 3577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:59743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:16:06.407039 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:40075
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:06.521535 3776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:06.521831 3776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:06.521878 3776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:06.526031 3776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:06.526093 3776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:06.526207 3776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:16:06.531795 3776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:06.533048 3776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:06.534152 3776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:06.544016 3782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:06.544037 3781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:06.544016 3784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.544363 3776 server_base.cc:1034] running on GCE node
I20260806 16:16:06.544885 3776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:06.545574 3776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:06.546766 3776 hybrid_clock.cc:648] HybridClock initialized: now 1786032966546727 us; error 51 us; skew 500 ppm
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:06.549669 3776 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:16:06.550640 3776 webserver.cc:460] Webserver started at http://127.18.90.2:35751/ using document root <none> and password file <none>
I20260806 16:16:06.551231 3776 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:06.551330 3776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:06.551570 3776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:06.553857 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "61b92e63740a46c2b5b18700db151ea9"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "4f7b0880ffd681fa0551f882ac0fb6d3"
server_key_iv: "1f9f4310e78fffcf7cb0d07b949ef237"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.554513 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "61b92e63740a46c2b5b18700db151ea9"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "4f7b0880ffd681fa0551f882ac0fb6d3"
server_key_iv: "1f9f4310e78fffcf7cb0d07b949ef237"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.558439 3776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:16:06.561033 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.562312 3776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:16:06.562461 3776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "61b92e63740a46c2b5b18700db151ea9"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "4f7b0880ffd681fa0551f882ac0fb6d3"
server_key_iv: "1f9f4310e78fffcf7cb0d07b949ef237"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.562594 3776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:06.573554 3776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:06.577165 3776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:06.577395 3776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:06.578027 3776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:06.578883 3776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:06.578958 3776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.579031 3776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:06.579077 3776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.588191 3776 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:38835
I20260806 16:16:06.588208 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:38835 every 8 connection(s)
I20260806 16:16:06.589164 3776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:16:06.594727 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3776
I20260806 16:16:06.594856 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:16:06.595155 18792 external_mini_cluster.cc:1428] Setting key 655122aad5fcabd02f7bd2a886259cf9
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:06.605724 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.591045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:48495 (local address 127.18.90.62:40075)
0806 16:16:06.591196 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.591200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.592138 (+ 938us) server_negotiation.cc:407] Connection header received
0806 16:16:06.593632 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.593635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.593700 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.593808 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:06.595142 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.595862 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.597282 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.597514 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.597626 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.601506 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:06.601511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:06.601514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:06.601568 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:06.603409 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.604009 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.604012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.604014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.604102 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.604548 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.604552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.604554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.604760 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:06.604875 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.605422 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.605552 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:16:06.606961 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.591438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:40075 (local address 127.18.90.2:48495)
0806 16:16:06.591986 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.592033 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.593407 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.594005 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.594012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.594379 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:06.594993 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.595010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.596003 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.596006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.597132 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.597142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.597686 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.597691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.597872 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.599510 (+ 1638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:06.599538 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:06.601307 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:06.603528 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:06.603534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:06.603551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:06.603896 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:06.604198 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:06.604202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:06.604204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:06.604441 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:06.604875 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:06.604884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:06.605161 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.605363 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.605722 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260806 16:16:06.608410 3908 heartbeater.cc:347] Connected to a master server at 127.18.90.62:40075
I20260806 16:16:06.608704 3908 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:16:06.609314 3908 heartbeater.cc:511] Master 127.18.90.62:40075 requested a full tablet report, sending...
I20260806 16:16:06.610502 3577 ts_manager.cc:194] Registered new tserver with Master: 61b92e63740a46c2b5b18700db151ea9 (127.18.90.2:38835)
I20260806 16:16:06.611366 3577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:48495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:16:06.625090 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:40075
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:06.740628 3913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:06.740895 3913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:06.740976 3913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:06.745265 3913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:06.745337 3913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:06.745448 3913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:16:06.750926 3913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:06.752090 3913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:06.753250 3913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:06.763124 3918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:06.763118 3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.763171 3913 server_base.cc:1034] running on GCE node
W20260806 16:16:06.763118 3921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:06.763676 3913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:06.764317 3913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:06.765488 3913 hybrid_clock.cc:648] HybridClock initialized: now 1786032966765454 us; error 46 us; skew 500 ppm
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:06.768402 3913 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:16:06.769268 3913 webserver.cc:460] Webserver started at http://127.18.90.3:42445/ using document root <none> and password file <none>
I20260806 16:16:06.769809 3913 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:06.769869 3913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:06.770063 3913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:06.772140 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "5b4f4c37c49841f58faf2da895feea46"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "648e57adfbc04333d2676914ea8fbd33"
server_key_iv: "316eb0f972e96f4cf404ada63636b9f7"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.772748 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "5b4f4c37c49841f58faf2da895feea46"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "648e57adfbc04333d2676914ea8fbd33"
server_key_iv: "316eb0f972e96f4cf404ada63636b9f7"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.776445 3913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:06.778811 3928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.780022 3913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:06.780158 3913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "5b4f4c37c49841f58faf2da895feea46"
format_stamp: "Formatted at 2026-08-06 16:16:06 on dist-test-slave-0b58"
server_key: "648e57adfbc04333d2676914ea8fbd33"
server_key_iv: "316eb0f972e96f4cf404ada63636b9f7"
server_key_version: "encryptionkey@0"
I20260806 16:16:06.780354 3913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:06.811872 3913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:06.814921 3913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:06.815138 3913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:06.815768 3913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:06.816706 3913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:06.816792 3913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.816854 3913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:06.816887 3913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:06.826699 3913 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:41865
I20260806 16:16:06.826719 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:41865 every 8 connection(s)
I20260806 16:16:06.827646 3913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:16:06.832854 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 3913
I20260806 16:16:06.833021 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:16:06.833364 18792 external_mini_cluster.cc:1428] Setting key 4ea47d87d1ea6919f84d433ec0a59719
Aug 06 16:16:06 dist-test-slave-0b58 krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:06.842439 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.829564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:39941 (local address 127.18.90.62:40075)
0806 16:16:06.829707 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.829712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.830581 (+ 869us) server_negotiation.cc:407] Connection header received
0806 16:16:06.831973 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.831977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.832037 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.832140 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:06.833808 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.834454 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.835940 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.836157 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.836239 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.838742 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:06.838746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:06.838748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:06.838777 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:06.840319 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.840845 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.840850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.840852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.840936 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.841302 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.841306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.841307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.841496 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:06.841606 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.842150 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.842272 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:16:06.843370 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.829953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:40075 (local address 127.18.90.3:39941)
0806 16:16:06.830428 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.830472 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.831753 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.832366 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.832379 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.832870 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:06.833587 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.833609 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.834588 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.834591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.835830 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.835841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.836338 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.836342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.836491 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.837048 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:06.837064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:06.838563 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:06.840440 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:06.840447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:06.840464 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:06.840737 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:06.841030 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:06.841033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:06.841035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:06.841184 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:06.841607 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:06.841615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:06.841911 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.842110 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.842463 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260806 16:16:06.844545 4043 heartbeater.cc:347] Connected to a master server at 127.18.90.62:40075
I20260806 16:16:06.844805 4043 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:06.845328 4043 heartbeater.cc:511] Master 127.18.90.62:40075 requested a full tablet report, sending...
I20260806 16:16:06.846457 3577 ts_manager.cc:194] Registered new tserver with Master: 5b4f4c37c49841f58faf2da895feea46 (127.18.90.3:41865)
I20260806 16:16:06.847062 3577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:39941
I20260806 16:16:06.847517 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:16:06.857044 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.849857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.18.90.62:40075)
0806 16:16:06.849986 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.849989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.850196 (+ 207us) server_negotiation.cc:407] Connection header received
0806 16:16:06.850286 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.850288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.850325 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.850392 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:06.851062 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.851553 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.852408 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.852628 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.852714 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.854114 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:06.854117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:06.854119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:06.854141 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:06.855613 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.855974 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.855978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.855979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.856050 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:06.856320 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:06.856328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:06.856329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:06.856544 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:06.856645 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.856833 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.856925 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:16:06.859849 3577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:16:06.863050 3577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:16:06.878293 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.872657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.18.90.1:35441)
0806 16:16:06.873214 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.873229 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.873253 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:16:06.873335 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.873342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.873553 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.874220 (+ 667us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:06.874369 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.875462 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.876983 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.877543 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.877673 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.877834 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.878012 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.878110 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":100,"threads_started":1}
I20260806 16:16:06.878284 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.872014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35441 (local address 127.0.0.1:54928)
0806 16:16:06.872539 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.872597 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.872744 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.873735 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.873741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.873775 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:06.874073 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.874080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.875595 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.875599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.876860 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.876868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.877674 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.877678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.877796 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.877834 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.877977 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.878110 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":179,"spinlock_wait_cycles":16128,"thread_start_us":102,"threads_started":1}
I20260806 16:16:06.878511 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.872210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:41865 (local address 127.0.0.1:50678)
0806 16:16:06.873103 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.873115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.873235 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.873984 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.873988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.874006 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:06.874125 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.874134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.875144 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.875151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.876563 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.876575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.878141 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.878145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.878222 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.878241 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.878308 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.878382 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":31,"thread_start_us":651,"threads_started":1}
I20260806 16:16:06.878613 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.871978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.18.90.2:38835)
0806 16:16:06.873043 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.873299 (+ 256us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.873322 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:16:06.873398 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.873402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.873552 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.873847 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:06.874209 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.875443 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.877002 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.877798 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.877939 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.878068 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.878356 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.878449 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":125,"threads_started":1}
I20260806 16:16:06.878683 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.871812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38835 (local address 127.0.0.1:40502)
0806 16:16:06.872539 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.872596 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.872746 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.873735 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.873742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.873776 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:06.874077 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.874085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.875637 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.875640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.876859 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.876870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.877943 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.877947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.878033 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.878055 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.878137 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.878210 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":105,"spinlock_wait_cycles":12544,"thread_start_us":96,"threads_started":1}
I20260806 16:16:06.879091 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.872519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.18.90.3:41865)
0806 16:16:06.873039 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.873053 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.873330 (+ 277us) server_negotiation.cc:407] Connection header received
0806 16:16:06.873412 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.873417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.873622 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.873787 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:06.874241 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.875009 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.876694 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.877538 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.877679 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.877837 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.878817 (+ 980us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.878937 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":123,"threads_started":1}
I20260806 16:16:06.881742 3976 tablet_service.cc:1467] Processing CreateTablet for tablet e40f2f6f5c664c23ba7c22224cd03e89 (DEFAULT_TABLE table=test-table [id=bbd96ec6d2f04ad9adc7f0209f160c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:06.881815 3841 tablet_service.cc:1467] Processing CreateTablet for tablet e40f2f6f5c664c23ba7c22224cd03e89 (DEFAULT_TABLE table=test-table [id=bbd96ec6d2f04ad9adc7f0209f160c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:06.882709 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40f2f6f5c664c23ba7c22224cd03e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:06.882728 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40f2f6f5c664c23ba7c22224cd03e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:06.884331 3704 tablet_service.cc:1467] Processing CreateTablet for tablet e40f2f6f5c664c23ba7c22224cd03e89 (DEFAULT_TABLE table=test-table [id=bbd96ec6d2f04ad9adc7f0209f160c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:06.885408 3704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40f2f6f5c664c23ba7c22224cd03e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:06.889703 4060 tablet_bootstrap.cc:492] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Bootstrap starting.
I20260806 16:16:06.890638 4061 tablet_bootstrap.cc:492] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Bootstrap starting.
I20260806 16:16:06.891989 4060 tablet_bootstrap.cc:654] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:06.892717 4061 tablet_bootstrap.cc:654] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:06.892877 4062 tablet_bootstrap.cc:492] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542: Bootstrap starting.
I20260806 16:16:06.893458 4060 log.cc:824] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:06.893991 4061 log.cc:824] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:06.895143 4062 tablet_bootstrap.cc:654] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:06.895720 4060 tablet_bootstrap.cc:492] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: No bootstrap required, opened a new log
I20260806 16:16:06.895876 4061 tablet_bootstrap.cc:492] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: No bootstrap required, opened a new log
I20260806 16:16:06.895869 4062 log.cc:824] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:06.895937 4060 ts_tablet_manager.cc:1397] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:06.896034 4061 ts_tablet_manager.cc:1397] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:06.897549 4062 tablet_bootstrap.cc:492] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542: No bootstrap required, opened a new log
I20260806 16:16:06.897764 4062 ts_tablet_manager.cc:1397] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 16:16:06.899961 4061 raft_consensus.cc:348] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.899961 4060 raft_consensus.cc:348] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.900292 4061 raft_consensus.cc:374] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:06.900290 4060 raft_consensus.cc:374] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:06.900364 4060 raft_consensus.cc:729] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4f4c37c49841f58faf2da895feea46, State: Initialized, Role: FOLLOWER
I20260806 16:16:06.900386 4061 raft_consensus.cc:729] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b92e63740a46c2b5b18700db151ea9, State: Initialized, Role: FOLLOWER
I20260806 16:16:06.900921 4061 consensus_queue.cc:260] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.900921 4060 consensus_queue.cc:260] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.901443 4062 raft_consensus.cc:348] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.901767 4062 raft_consensus.cc:374] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:06.901872 4062 raft_consensus.cc:729] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f58e711e6a40e08b33997db3e5b542, State: Initialized, Role: FOLLOWER
I20260806 16:16:06.902031 4061 ts_tablet_manager.cc:1428] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:16:06.902031 4060 ts_tablet_manager.cc:1428] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:16:06.902457 3908 heartbeater.cc:503] Master 127.18.90.62:40075 was elected leader, sending a full tablet report...
I20260806 16:16:06.902457 4043 heartbeater.cc:503] Master 127.18.90.62:40075 was elected leader, sending a full tablet report...
I20260806 16:16:06.902572 4062 consensus_queue.cc:260] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.903724 4062 ts_tablet_manager.cc:1428] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260806 16:16:06.904143 3770 heartbeater.cc:503] Master 127.18.90.62:40075 was elected leader, sending a full tablet report...
I20260806 16:16:06.963253 4068 raft_consensus.cc:482] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:06.963450 4068 raft_consensus.cc:504] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.964600 4068 leader_election.cc:288] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b92e63740a46c2b5b18700db151ea9 (127.18.90.2:38835), d5f58e711e6a40e08b33997db3e5b542 (127.18.90.1:35441)
I20260806 16:16:06.968737 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.964912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:46405 (local address 127.18.90.2:38835)
0806 16:16:06.965039 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.965045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.965231 (+ 186us) server_negotiation.cc:407] Connection header received
0806 16:16:06.965305 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.965308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.965357 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.965432 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:06.965750 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.966435 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.967721 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.968155 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.968235 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.968353 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.968535 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.968609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:16:06.968737 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.964838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:38835 (local address 127.18.90.3:46405)
0806 16:16:06.965037 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.965054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.965156 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.965464 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.965467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.965484 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:06.965618 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.965625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.966547 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.966550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.967601 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.967611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.968295 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.968298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.968404 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.968428 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.968495 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.968584 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":29}
I20260806 16:16:06.968770 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.965018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:49343 (local address 127.18.90.1:35441)
0806 16:16:06.965158 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.965163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.965323 (+ 160us) server_negotiation.cc:407] Connection header received
0806 16:16:06.965472 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.965475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.965520 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:06.965601 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:06.965956 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.966534 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.967779 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.968177 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.968321 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.968384 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:06.968585 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 16:16:06.968657 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:16:06.968751 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.964915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35441 (local address 127.18.90.3:49343)
0806 16:16:06.965223 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:06.965240 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:06.965341 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:06.965622 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:06.965624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:06.965638 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:06.965782 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.965788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.966647 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.966650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.967623 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:06.967634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:06.968324 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:06.968327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:06.968439 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:06.968456 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:16:06.968551 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 16:16:06.968611 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":42,"thread_start_us":133,"threads_started":1}
I20260806 16:16:06.969660 3724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f2f6f5c664c23ba7c22224cd03e89" candidate_uuid: "5b4f4c37c49841f58faf2da895feea46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f58e711e6a40e08b33997db3e5b542" is_pre_election: true
I20260806 16:16:06.969660 3861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f2f6f5c664c23ba7c22224cd03e89" candidate_uuid: "5b4f4c37c49841f58faf2da895feea46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b92e63740a46c2b5b18700db151ea9" is_pre_election: true
I20260806 16:16:06.969996 3724 raft_consensus.cc:2454] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b4f4c37c49841f58faf2da895feea46 in term 0.
I20260806 16:16:06.969996 3861 raft_consensus.cc:2454] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b4f4c37c49841f58faf2da895feea46 in term 0.
I20260806 16:16:06.970476 3932 leader_election.cc:302] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b4f4c37c49841f58faf2da895feea46, d5f58e711e6a40e08b33997db3e5b542; no voters:
I20260806 16:16:06.970813 4068 raft_consensus.cc:2781] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:16:06.970949 4068 raft_consensus.cc:482] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:16:06.971061 4068 raft_consensus.cc:3037] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:06.972105 4068 raft_consensus.cc:504] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.972487 4068 leader_election.cc:288] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [CANDIDATE]: Term 1 election: Requested vote from peers 61b92e63740a46c2b5b18700db151ea9 (127.18.90.2:38835), d5f58e711e6a40e08b33997db3e5b542 (127.18.90.1:35441)
I20260806 16:16:06.972867 3724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f2f6f5c664c23ba7c22224cd03e89" candidate_uuid: "5b4f4c37c49841f58faf2da895feea46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f58e711e6a40e08b33997db3e5b542"
I20260806 16:16:06.972869 3861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f2f6f5c664c23ba7c22224cd03e89" candidate_uuid: "5b4f4c37c49841f58faf2da895feea46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b92e63740a46c2b5b18700db151ea9"
I20260806 16:16:06.973023 3861 raft_consensus.cc:3037] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:06.973023 3724 raft_consensus.cc:3037] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:06.974359 3861 raft_consensus.cc:2454] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b4f4c37c49841f58faf2da895feea46 in term 1.
I20260806 16:16:06.974359 3724 raft_consensus.cc:2454] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b4f4c37c49841f58faf2da895feea46 in term 1.
I20260806 16:16:06.974702 3930 leader_election.cc:302] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b4f4c37c49841f58faf2da895feea46, 61b92e63740a46c2b5b18700db151ea9; no voters:
I20260806 16:16:06.974920 4068 raft_consensus.cc:2781] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:06.975198 4068 raft_consensus.cc:686] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4f4c37c49841f58faf2da895feea46, State: Running, Role: LEADER
I20260806 16:16:06.975744 4068 consensus_queue.cc:237] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } }
I20260806 16:16:06.979148 3576 catalog_manager.cc:5495] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b4f4c37c49841f58faf2da895feea46 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "5b4f4c37c49841f58faf2da895feea46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b4f4c37c49841f58faf2da895feea46" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 41865 } health_report { overall_health: HEALTHY } } }
I20260806 16:16:07.003532 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:06.999649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.18.90.3:41865)
0806 16:16:06.999801 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:06.999806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:06.999823 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:16:06.999926 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:06.999929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:06.999986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.000101 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:07.000422 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.000925 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.001846 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.002114 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.002228 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.002606 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:07.003091 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:07.003239 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.003280 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.003341 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:16:07.009748 3861 raft_consensus.cc:1261] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 1 FOLLOWER]: Refusing update from remote peer 5b4f4c37c49841f58faf2da895feea46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:07.009748 3724 raft_consensus.cc:1261] T e40f2f6f5c664c23ba7c22224cd03e89 P d5f58e711e6a40e08b33997db3e5b542 [term 1 FOLLOWER]: Refusing update from remote peer 5b4f4c37c49841f58faf2da895feea46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:07.010499 4071 consensus_queue.cc:1035] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b92e63740a46c2b5b18700db151ea9" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:07.010744 4068 consensus_queue.cc:1035] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f58e711e6a40e08b33997db3e5b542" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:07.017999 4074 mvcc.cc:205] Tried to move back new op lower bound from 7315591032863645696 to 7315591032735694848. Current Snapshot: MvccSnapshot[applied={T|T < 7315591032863645696}]
I20260806 16:16:07.018733 4075 mvcc.cc:205] Tried to move back new op lower bound from 7315591032863645696 to 7315591032735694848. Current Snapshot: MvccSnapshot[applied={T|T < 7315591032863645696}]
I20260806 16:16:07.020457 4076 mvcc.cc:205] Tried to move back new op lower bound from 7315591032863645696 to 7315591032735694848. Current Snapshot: MvccSnapshot[applied={T|T < 7315591032863645696}]
I20260806 16:16:07.027655 3576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:07.027834 3576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:07.030584 3576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d36c4d64681f40a0b9dcbf22c1d0bb3a: Sending DeleteTablet for 3 replicas of tablet e40f2f6f5c664c23ba7c22224cd03e89
I20260806 16:16:07.031440 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet e40f2f6f5c664c23ba7c22224cd03e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:40502
I20260806 16:16:07.031459 3704 tablet_service.cc:1514] Processing DeleteTablet for tablet e40f2f6f5c664c23ba7c22224cd03e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:54928
I20260806 16:16:07.031548 3976 tablet_service.cc:1514] Processing DeleteTablet for tablet e40f2f6f5c664c23ba7c22224cd03e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:07 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:50678
I20260806 16:16:07.032200 4083 tablet_replica.cc:330] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: stopping tablet replica
I20260806 16:16:07.032578 4083 raft_consensus.cc:2229] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:07.032752 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3641
I20260806 16:16:07.032946 4083 raft_consensus.cc:2258] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:07.033450 4084 tablet_replica.cc:330] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: stopping tablet replica
I20260806 16:16:07.033793 4084 raft_consensus.cc:2229] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:16:07.034190 4084 raft_consensus.cc:2258] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:07.036824 4083 ts_tablet_manager.cc:1905] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:07.037230 4084 ts_tablet_manager.cc:1905] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:07.039958 4084 ts_tablet_manager.cc:1918] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:07.040135 4084 log.cc:1196] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/e40f2f6f5c664c23ba7c22224cd03e89
I20260806 16:16:07.040477 4083 ts_tablet_manager.cc:1918] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:07.040627 4083 log.cc:1196] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/e40f2f6f5c664c23ba7c22224cd03e89
I20260806 16:16:07.040653 4084 ts_tablet_manager.cc:1939] T e40f2f6f5c664c23ba7c22224cd03e89 P 5b4f4c37c49841f58faf2da895feea46: Deleting consensus metadata
I20260806 16:16:07.041002 4083 ts_tablet_manager.cc:1939] T e40f2f6f5c664c23ba7c22224cd03e89 P 61b92e63740a46c2b5b18700db151ea9: Deleting consensus metadata
I20260806 16:16:07.042366 3561 catalog_manager.cc:4841] TS 5b4f4c37c49841f58faf2da895feea46 (127.18.90.3:41865): tablet e40f2f6f5c664c23ba7c22224cd03e89 (table test-table [id=bbd96ec6d2f04ad9adc7f0209f160c67]) successfully deleted
W20260806 16:16:07.042492 3564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:35441 (error 108)
I20260806 16:16:07.042593 3562 catalog_manager.cc:4841] TS 61b92e63740a46c2b5b18700db151ea9 (127.18.90.2:38835): tablet e40f2f6f5c664c23ba7c22224cd03e89 (table test-table [id=bbd96ec6d2f04ad9adc7f0209f160c67]) successfully deleted
I20260806 16:16:07.042634 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3776
I20260806 16:16:07.044792 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.044001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35441 (local address 127.0.0.1:54936)
0806 16:16:07.044604 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.044686 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:35441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487}
W20260806 16:16:07.045637 3564 catalog_manager.cc:4584] TS d5f58e711e6a40e08b33997db3e5b542 (127.18.90.1:35441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e40f2f6f5c664c23ba7c22224cd03e89: Network error: Client connection negotiation failed: client connection to 127.18.90.1:35441: connect: Connection refused (error 111)
I20260806 16:16:07.051537 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3913
I20260806 16:16:07.058328 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 3545
2026-08-06T16:16:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2328 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T16:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:07Z Disabled control of system clock
I20260806 16:16:07.091035 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:35177
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:35177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:16:07.205458 4090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:07.205744 4090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:07.205780 4090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:07.209916 4090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:07.209983 4090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:07.209998 4090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:07.210016 4090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:07.210031 4090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:07.214874 4090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:35177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:07.216009 4090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:07.217495 4090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:16:07.226178 4090 server_base.cc:1034] running on GCE node
W20260806 16:16:07.226112 4098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:07.226112 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:07.226111 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:07.227203 4090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:07.228308 4090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:07.229485 4090 hybrid_clock.cc:648] HybridClock initialized: now 1786032967229472 us; error 25 us; skew 500 ppm
I20260806 16:16:07.231310 4090 webserver.cc:460] Webserver started at http://127.18.90.62:33361/ using document root <none> and password file <none>
I20260806 16:16:07.231943 4090 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:07.232007 4090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:07.232249 4090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:07.234390 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "5e04947d97144ee4aa265979c78eef18"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "052ad0e7f1b20608ec0d23d8e276f8b8"
server_key_iv: "bb7359bf1b33fef529e986733a92cc25"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.234975 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "5e04947d97144ee4aa265979c78eef18"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "052ad0e7f1b20608ec0d23d8e276f8b8"
server_key_iv: "bb7359bf1b33fef529e986733a92cc25"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.238690 4090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:16:07.241189 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.242467 4090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:07.242604 4090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "5e04947d97144ee4aa265979c78eef18"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "052ad0e7f1b20608ec0d23d8e276f8b8"
server_key_iv: "bb7359bf1b33fef529e986733a92cc25"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.242714 4090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:07.259339 4090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:07.260061 4090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:07.260316 4090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:07.267642 4090 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:35177
I20260806 16:16:07.267678 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:35177 every 8 connection(s)
I20260806 16:16:07.268671 4090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:07.271809 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:07.277407 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18: Bootstrap starting.
I20260806 16:16:07.277535 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4090
I20260806 16:16:07.277655 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:07.277971 18792 external_mini_cluster.cc:1428] Setting key 2f00facddb982c22c62709f2c85cd292
I20260806 16:16:07.280236 4159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:07.281039 4159 log.cc:824] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:07.283097 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18: No bootstrap required, opened a new log
I20260806 16:16:07.287405 4159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e04947d97144ee4aa265979c78eef18" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 35177 } }
I20260806 16:16:07.287583 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.279401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.18.90.62:35177)
0806 16:16:07.280047 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.280063 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.280097 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:16:07.280781 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.280806 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.281298 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.281660 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:07.282085 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.283047 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.284350 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.284814 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.284927 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.285819 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:07.285824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:07.285836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:07.286239 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:07.286293 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:07.286312 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:07.286387 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.286583 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.286869 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260806 16:16:07.287659 4159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:07.287745 4159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e04947d97144ee4aa265979c78eef18, State: Initialized, Role: FOLLOWER
I20260806 16:16:07.288268 4159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e04947d97144ee4aa265979c78eef18" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 35177 } }
I20260806 16:16:07.288426 4159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:07.288486 4159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:07.288575 4159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:07.289445 4159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e04947d97144ee4aa265979c78eef18" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 35177 } }
I20260806 16:16:07.289831 4159 leader_election.cc:302] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e04947d97144ee4aa265979c78eef18; no voters:
I20260806 16:16:07.290189 4159 leader_election.cc:288] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:07.290298 4164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:07.290535 4164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [term 1 LEADER]: Becoming Leader. State: Replica: 5e04947d97144ee4aa265979c78eef18, State: Running, Role: LEADER
I20260806 16:16:07.290863 4164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e04947d97144ee4aa265979c78eef18" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 35177 } }
I20260806 16:16:07.291484 4159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:07.292701 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e04947d97144ee4aa265979c78eef18. Latest consensus state: current_term: 1 leader_uuid: "5e04947d97144ee4aa265979c78eef18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e04947d97144ee4aa265979c78eef18" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 35177 } } }
I20260806 16:16:07.292833 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:07.292681 4165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e04947d97144ee4aa265979c78eef18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e04947d97144ee4aa265979c78eef18" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 35177 } } }
I20260806 16:16:07.292913 4165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:07.293216 4173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:07.296078 4173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:07.301352 4173 catalog_manager.cc:1347] Generated new cluster ID: a8f1c40ee7de487cad7e03abeeeb2c20
I20260806 16:16:07.301420 4173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:07.316087 4173 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:07.317386 4173 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:07.325560 4173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18: Generated new TSK 0
I20260806 16:16:07.326303 4173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:07.336695 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:35177
--builtin_ntp_servers=127.18.90.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:16:07.449831 4183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:07.450105 4183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:07.450179 4183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:07.454134 4183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:07.454214 4183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:07.454298 4183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:16:07.458958 4183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:35177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:07.460367 4183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:07.461781 4183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:07.471109 4189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:07.471217 4191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:07.471112 4188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:07.471491 4183 server_base.cc:1034] running on GCE node
I20260806 16:16:07.471875 4183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:07.472515 4183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:07.473680 4183 hybrid_clock.cc:648] HybridClock initialized: now 1786032967473659 us; error 32 us; skew 500 ppm
I20260806 16:16:07.475392 4183 webserver.cc:460] Webserver started at http://127.18.90.1:33491/ using document root <none> and password file <none>
I20260806 16:16:07.475924 4183 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:07.475980 4183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:07.476137 4183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:07.478225 4183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "a467d4bdc7114ea3ac40cfc0386a171e"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "4f2c2894fc65cecba608b36de57cfbab"
server_key_iv: "9b82382ca726b9168902a2dab4f2ed7d"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.478724 4183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "a467d4bdc7114ea3ac40cfc0386a171e"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "4f2c2894fc65cecba608b36de57cfbab"
server_key_iv: "9b82382ca726b9168902a2dab4f2ed7d"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.482213 4183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:16:07.484448 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.485605 4183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:07.485736 4183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "a467d4bdc7114ea3ac40cfc0386a171e"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "4f2c2894fc65cecba608b36de57cfbab"
server_key_iv: "9b82382ca726b9168902a2dab4f2ed7d"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.485844 4183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:07.500077 4183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:07.500773 4183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:07.500975 4183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:07.501551 4183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:07.502454 4183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:07.502525 4183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.502612 4183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:07.502676 4183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.511277 4183 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:38923
I20260806 16:16:07.511314 4312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:38923 every 8 connection(s)
I20260806 16:16:07.512183 4183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:16:07.513787 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4183
I20260806 16:16:07.513911 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:16:07.514261 18792 external_mini_cluster.cc:1428] Setting key 650602bed64fe4e18c229947cf56d181
I20260806 16:16:07.516311 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:35177
--builtin_ntp_servers=127.18.90.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:16:07.522773 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.514162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:38525 (local address 127.18.90.62:35177)
0806 16:16:07.514327 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.514334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.515480 (+ 1146us) server_negotiation.cc:407] Connection header received
0806 16:16:07.517063 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.517066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.517122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.517203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:07.518264 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.518797 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.520607 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.520861 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.520969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.522287 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:07.522290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:07.522292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:07.522304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:07.522323 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:07.522330 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:07.522389 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.522592 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.522656 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:16:07.524102 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.514607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:35177 (local address 127.18.90.1:38525)
0806 16:16:07.515345 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.515397 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:07.516848 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:07.517369 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:07.517375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:07.517460 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:07.518085 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.518107 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.518939 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.518945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.520479 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.520494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.521052 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.521059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.521298 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.521973 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:07.521975 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:07.522035 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:07.522129 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:07.522423 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:07.522435 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:07.522442 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:16:07.522581 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 16:16:07.522938 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260806 16:16:07.525484 4313 heartbeater.cc:347] Connected to a master server at 127.18.90.62:35177
I20260806 16:16:07.525731 4313 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:07.526278 4313 heartbeater.cc:511] Master 127.18.90.62:35177 requested a full tablet report, sending...
I20260806 16:16:07.527791 4122 ts_manager.cc:194] Registered new tserver with Master: a467d4bdc7114ea3ac40cfc0386a171e (127.18.90.1:38923)
I20260806 16:16:07.528982 4122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.1:38525
W20260806 16:16:07.629320 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:07.629642 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:07.629715 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:07.633810 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:07.633901 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:07.633997 4316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:16:07.639088 4316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:35177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:07.640239 4316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:07.641739 4316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:07.652115 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:07.652107 4321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:07.652184 4316 server_base.cc:1034] running on GCE node
W20260806 16:16:07.652103 4322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:07.652805 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:07.653417 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:07.654577 4316 hybrid_clock.cc:648] HybridClock initialized: now 1786032967654561 us; error 26 us; skew 500 ppm
I20260806 16:16:07.656430 4316 webserver.cc:460] Webserver started at http://127.18.90.2:37783/ using document root <none> and password file <none>
I20260806 16:16:07.657024 4316 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:07.657087 4316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:07.657346 4316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:07.659420 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "5991b0e2c5414799866035d84d154cb3"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "003ba478b36dec0cb8c2896406019e6d"
server_key_iv: "642216b4cbfd590e5d281e94379569ec"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.659974 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "5991b0e2c5414799866035d84d154cb3"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "003ba478b36dec0cb8c2896406019e6d"
server_key_iv: "642216b4cbfd590e5d281e94379569ec"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.663635 4316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260806 16:16:07.666076 4331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.667337 4316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:16:07.667456 4316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "5991b0e2c5414799866035d84d154cb3"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "003ba478b36dec0cb8c2896406019e6d"
server_key_iv: "642216b4cbfd590e5d281e94379569ec"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.667565 4316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:07.680914 4316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:07.681571 4316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:07.681763 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:07.682353 4316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:07.683257 4316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:07.683306 4316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.683344 4316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:07.683401 4316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.692509 4316 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44107
I20260806 16:16:07.692521 4445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44107 every 8 connection(s)
I20260806 16:16:07.693619 4316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:16:07.702229 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.695408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:46385 (local address 127.18.90.62:35177)
0806 16:16:07.695546 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.695549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.696605 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 16:16:07.697949 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.697951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.697999 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.698071 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:07.698915 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.699440 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.700547 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.700753 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.700835 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.701830 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:07.701833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:07.701835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:07.701846 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:07.701865 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:07.701872 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:07.701928 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.702065 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.702112 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:16:07.703090 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.695801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:35177 (local address 127.18.90.2:46385)
0806 16:16:07.696458 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.696498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:07.697762 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:07.698231 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:07.698237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:07.698318 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:07.698793 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.698805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.699542 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.699544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.700438 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.700448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.700880 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.700883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.701020 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.701589 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:07.701591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:07.701630 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:07.701695 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:07.701942 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:07.701949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:07.701953 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:16:07.702040 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 16:16:07.702251 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260806 16:16:07.703624 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4316
I20260806 16:16:07.703765 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:16:07.704048 18792 external_mini_cluster.cc:1428] Setting key 2a118e529947c62692e8a34e2c2bb447
I20260806 16:16:07.704349 4446 heartbeater.cc:347] Connected to a master server at 127.18.90.62:35177
I20260806 16:16:07.704577 4446 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:07.705192 4446 heartbeater.cc:511] Master 127.18.90.62:35177 requested a full tablet report, sending...
I20260806 16:16:07.706223 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:35177
--builtin_ntp_servers=127.18.90.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:16:07.706274 4122 ts_manager.cc:194] Registered new tserver with Master: 5991b0e2c5414799866035d84d154cb3 (127.18.90.2:44107)
I20260806 16:16:07.706844 4122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.2:46385
W20260806 16:16:07.818452 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:07.818711 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:07.818748 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:07.823982 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:07.824044 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:07.824100 4449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:16:07.828963 4449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:35177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:07.829995 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:07.831286 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:07.840574 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:07.840658 4449 server_base.cc:1034] running on GCE node
W20260806 16:16:07.840582 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:07.840582 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:07.841235 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:07.841825 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:07.843009 4449 hybrid_clock.cc:648] HybridClock initialized: now 1786032967842960 us; error 59 us; skew 500 ppm
I20260806 16:16:07.844952 4449 webserver.cc:460] Webserver started at http://127.18.90.3:39107/ using document root <none> and password file <none>
I20260806 16:16:07.845557 4449 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:07.845655 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:07.845896 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:07.847971 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "88f2942ee6024d59983801b398aac016"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "cb6082ae5c50cc03b96d47301f6fad51"
server_key_iv: "de9b47f174794329f31839e10fa0cd7e"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.848596 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "88f2942ee6024d59983801b398aac016"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "cb6082ae5c50cc03b96d47301f6fad51"
server_key_iv: "de9b47f174794329f31839e10fa0cd7e"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.852157 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:16:07.854558 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.855755 4449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:07.855901 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "88f2942ee6024d59983801b398aac016"
format_stamp: "Formatted at 2026-08-06 16:16:07 on dist-test-slave-0b58"
server_key: "cb6082ae5c50cc03b96d47301f6fad51"
server_key_iv: "de9b47f174794329f31839e10fa0cd7e"
server_key_version: "encryptionkey@0"
I20260806 16:16:07.856016 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:07.871990 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:07.872656 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:07.872849 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:07.873416 4449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:07.874361 4449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:07.874431 4449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.874522 4449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:07.874573 4449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:07.883198 4449 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:45527
I20260806 16:16:07.883219 4578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:45527 every 8 connection(s)
I20260806 16:16:07.884052 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:16:07.893052 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4449
I20260806 16:16:07.893169 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:16:07.893347 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.885855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:57819 (local address 127.18.90.62:35177)
0806 16:16:07.885998 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.886002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.887028 (+ 1026us) server_negotiation.cc:407] Connection header received
0806 16:16:07.888466 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.888469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.888528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.888631 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:07.889574 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.890182 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.891362 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.891601 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.891732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.892861 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:07.892864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:07.892866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:07.892878 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:07.892905 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:07.892912 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:07.892985 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.893147 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.893206 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:16:07.893553 18792 external_mini_cluster.cc:1428] Setting key e14aa884767ae62993476d1a3545877b
I20260806 16:16:07.894322 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.886264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:35177 (local address 127.18.90.3:57819)
0806 16:16:07.886890 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.886928 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:07.888208 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:07.888814 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:07.888820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:07.888959 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:07.889429 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.889445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.890300 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.890303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.891248 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.891257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.891783 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.891787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.891956 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.892596 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:07.892598 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:07.892634 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:07.892694 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:07.892999 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:07.893010 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:07.893017 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:16:07.893140 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 16:16:07.893407 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260806 16:16:07.895427 4579 heartbeater.cc:347] Connected to a master server at 127.18.90.62:35177
I20260806 16:16:07.895665 4579 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:07.896157 4579 heartbeater.cc:511] Master 127.18.90.62:35177 requested a full tablet report, sending...
I20260806 16:16:07.897214 4121 ts_manager.cc:194] Registered new tserver with Master: 88f2942ee6024d59983801b398aac016 (127.18.90.3:45527)
I20260806 16:16:07.897799 4121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.3:57819
I20260806 16:16:07.907420 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:16:07.913353 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.909506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.18.90.62:35177)
0806 16:16:07.909676 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.909681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.909829 (+ 148us) server_negotiation.cc:407] Connection header received
0806 16:16:07.909998 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.910001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.910041 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.910121 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:07.910400 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.910902 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.911738 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.911951 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.912059 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.912930 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:07.912934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:07.912937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:07.912956 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:07.912973 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:07.912980 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:07.913058 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.913148 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.913206 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 16:16:07.916086 4121 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:16:07.919149 4121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:16:07.933640 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.927841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.18.90.3:45527)
0806 16:16:07.928364 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.928376 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.929040 (+ 664us) server_negotiation.cc:407] Connection header received
0806 16:16:07.929128 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.929134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.929362 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.929544 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:07.929792 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.930660 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.932305 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.932943 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.933078 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.933304 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.933372 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.933471 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":87,"threads_started":1}
I20260806 16:16:07.933633 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.927645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:45527 (local address 127.0.0.1:51866)
0806 16:16:07.928832 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.928846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:07.928936 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:07.929515 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:07.929518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:07.929537 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:07.929635 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.929653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.930777 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.930783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.932125 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.932136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.933079 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.933083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.933194 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.933233 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 16:16:07.933335 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 16:16:07.933460 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"thread_start_us":115,"threads_started":1}
I20260806 16:16:07.933810 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.927548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:38923 (local address 127.0.0.1:57304)
0806 16:16:07.928027 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.928074 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:07.928199 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:07.929350 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:07.929357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:07.929382 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:07.929529 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.929536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.931185 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.931188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.932308 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.932316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.933405 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.933409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.933494 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.933516 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:07.933567 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 16:16:07.933658 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":101,"threads_started":1}
I20260806 16:16:07.934001 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.928406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44107 (local address 127.0.0.1:59880)
0806 16:16:07.929038 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:07.929050 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:07.929119 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:07.929817 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:07.929820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:07.929841 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:07.929916 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.929921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.931232 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.931234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.932357 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:07.932368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.933629 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.933632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:07.933706 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.933726 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 16:16:07.933793 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 16:16:07.933866 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":106,"threads_started":1}
I20260806 16:16:07.934463 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.928652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.18.90.2:44107)
0806 16:16:07.929377 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.929389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.929411 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:16:07.929479 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.929484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.929679 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.929828 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:07.930005 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.931102 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.932481 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.933260 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.933381 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.933570 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.934195 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.934308 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":89,"threads_started":1}
I20260806 16:16:07.935189 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:07.928426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.18.90.1:38923)
0806 16:16:07.928809 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:07.928821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:07.928848 (+ 27us) server_negotiation.cc:407] Connection header received
0806 16:16:07.928931 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:07.928937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:07.929154 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:07.929684 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:07.929845 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.931039 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.932438 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:07.933260 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:07.934686 (+ 1426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:07.934870 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:07.934921 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 16:16:07.935024 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260806 16:16:07.937101 4511 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9bf840b39b45deb571ff64ea6e32f9 (DEFAULT_TABLE table=test-table [id=e4b55b3e14de47d8b4e4fd5ef8e880aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:07.937400 4379 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9bf840b39b45deb571ff64ea6e32f9 (DEFAULT_TABLE table=test-table [id=e4b55b3e14de47d8b4e4fd5ef8e880aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:07.937726 4246 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9bf840b39b45deb571ff64ea6e32f9 (DEFAULT_TABLE table=test-table [id=e4b55b3e14de47d8b4e4fd5ef8e880aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:07.938205 4511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9bf840b39b45deb571ff64ea6e32f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:07.938479 4379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9bf840b39b45deb571ff64ea6e32f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:07.938649 4246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9bf840b39b45deb571ff64ea6e32f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:07.944819 4596 tablet_bootstrap.cc:492] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Bootstrap starting.
I20260806 16:16:07.946156 4597 tablet_bootstrap.cc:492] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: Bootstrap starting.
I20260806 16:16:07.946816 4598 tablet_bootstrap.cc:492] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Bootstrap starting.
I20260806 16:16:07.947369 4596 tablet_bootstrap.cc:654] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:07.948133 4596 log.cc:824] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:07.948423 4597 tablet_bootstrap.cc:654] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:07.948937 4598 tablet_bootstrap.cc:654] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:07.949954 4598 log.cc:824] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:07.950670 4596 tablet_bootstrap.cc:492] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: No bootstrap required, opened a new log
I20260806 16:16:07.950893 4596 ts_tablet_manager.cc:1397] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:07.950954 4597 log.cc:824] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:07.952142 4598 tablet_bootstrap.cc:492] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: No bootstrap required, opened a new log
I20260806 16:16:07.952373 4598 ts_tablet_manager.cc:1397] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:16:07.953100 4597 tablet_bootstrap.cc:492] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: No bootstrap required, opened a new log
I20260806 16:16:07.953298 4597 ts_tablet_manager.cc:1397] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260806 16:16:07.954666 4596 raft_consensus.cc:348] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:07.954918 4596 raft_consensus.cc:374] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:07.954972 4596 raft_consensus.cc:729] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f2942ee6024d59983801b398aac016, State: Initialized, Role: FOLLOWER
I20260806 16:16:07.955418 4596 consensus_queue.cc:260] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:07.956537 4596 ts_tablet_manager.cc:1428] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260806 16:16:07.956777 4598 raft_consensus.cc:348] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:07.957058 4598 raft_consensus.cc:374] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:07.957125 4598 raft_consensus.cc:729] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5991b0e2c5414799866035d84d154cb3, State: Initialized, Role: FOLLOWER
I20260806 16:16:07.957334 4579 heartbeater.cc:503] Master 127.18.90.62:35177 was elected leader, sending a full tablet report...
I20260806 16:16:07.957587 4598 consensus_queue.cc:260] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:07.957613 4597 raft_consensus.cc:348] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:07.957867 4597 raft_consensus.cc:374] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:07.957937 4597 raft_consensus.cc:729] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a467d4bdc7114ea3ac40cfc0386a171e, State: Initialized, Role: FOLLOWER
I20260806 16:16:07.958335 4446 heartbeater.cc:503] Master 127.18.90.62:35177 was elected leader, sending a full tablet report...
I20260806 16:16:07.958377 4597 consensus_queue.cc:260] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:07.958652 4598 ts_tablet_manager.cc:1428] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:16:07.959256 4597 ts_tablet_manager.cc:1428] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:16:07.959637 4313 heartbeater.cc:503] Master 127.18.90.62:35177 was elected leader, sending a full tablet report...
W20260806 16:16:08.013393 4314 tablet.cc:2369] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:16:08.036271 4603 raft_consensus.cc:482] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:08.036527 4603 raft_consensus.cc:504] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:08.037840 4603 leader_election.cc:288] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a467d4bdc7114ea3ac40cfc0386a171e (127.18.90.1:38923), 88f2942ee6024d59983801b398aac016 (127.18.90.3:45527)
I20260806 16:16:08.041558 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:08.038159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:45267 (local address 127.18.90.1:38923)
0806 16:16:08.038288 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:08.038292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:08.038375 (+ 83us) server_negotiation.cc:407] Connection header received
0806 16:16:08.038551 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:08.038553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:08.038602 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:08.038692 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:08.038954 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.039471 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.040587 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.040985 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.041104 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:08.041179 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:08.041353 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 16:16:08.041438 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:16:08.041559 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:08.038060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:38923 (local address 127.18.90.2:45267)
0806 16:16:08.038265 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:08.038286 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:08.038402 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:08.038704 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:08.038707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:08.038725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:08.038842 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:08.038849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.039585 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.039589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:08.040459 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:08.040469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.041111 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.041115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:08.041219 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:08.041245 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 16:16:08.041334 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:16:08.041420 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 16:16:08.041790 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:08.038175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:45527 (local address 127.18.90.2:42843)
0806 16:16:08.038567 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:08.038578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:08.038707 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:08.038977 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:08.038980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:08.038995 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:08.039122 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:08.039128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.039852 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.039857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:08.040704 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:08.040713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.041356 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.041359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:08.041477 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:08.041496 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:16:08.041576 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:16:08.041642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":135,"threads_started":1}
I20260806 16:16:08.041779 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:08.038314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:42843 (local address 127.18.90.3:45527)
0806 16:16:08.038490 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:08.038495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:08.038661 (+ 166us) server_negotiation.cc:407] Connection header received
0806 16:16:08.038829 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:08.038836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:08.038877 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:08.038966 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:08.039236 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.039735 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.040826 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.041222 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.041342 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:08.041409 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:08.041606 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 16:16:08.041677 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 16:16:08.042431 4266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9bf840b39b45deb571ff64ea6e32f9" candidate_uuid: "5991b0e2c5414799866035d84d154cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" is_pre_election: true
I20260806 16:16:08.042470 4531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9bf840b39b45deb571ff64ea6e32f9" candidate_uuid: "5991b0e2c5414799866035d84d154cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f2942ee6024d59983801b398aac016" is_pre_election: true
I20260806 16:16:08.042742 4266 raft_consensus.cc:2454] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5991b0e2c5414799866035d84d154cb3 in term 0.
I20260806 16:16:08.042752 4531 raft_consensus.cc:2454] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5991b0e2c5414799866035d84d154cb3 in term 0.
I20260806 16:16:08.043236 4335 leader_election.cc:302] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5991b0e2c5414799866035d84d154cb3, 88f2942ee6024d59983801b398aac016; no voters:
I20260806 16:16:08.043522 4603 raft_consensus.cc:2781] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:16:08.043607 4603 raft_consensus.cc:482] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:16:08.043663 4603 raft_consensus.cc:3037] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:08.044814 4603 raft_consensus.cc:504] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:08.045230 4603 leader_election.cc:288] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [CANDIDATE]: Term 1 election: Requested vote from peers a467d4bdc7114ea3ac40cfc0386a171e (127.18.90.1:38923), 88f2942ee6024d59983801b398aac016 (127.18.90.3:45527)
I20260806 16:16:08.045570 4266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9bf840b39b45deb571ff64ea6e32f9" candidate_uuid: "5991b0e2c5414799866035d84d154cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a467d4bdc7114ea3ac40cfc0386a171e"
I20260806 16:16:08.045674 4531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9bf840b39b45deb571ff64ea6e32f9" candidate_uuid: "5991b0e2c5414799866035d84d154cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f2942ee6024d59983801b398aac016"
I20260806 16:16:08.045737 4266 raft_consensus.cc:3037] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:08.045849 4531 raft_consensus.cc:3037] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:08.046941 4266 raft_consensus.cc:2454] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5991b0e2c5414799866035d84d154cb3 in term 1.
I20260806 16:16:08.047052 4531 raft_consensus.cc:2454] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5991b0e2c5414799866035d84d154cb3 in term 1.
I20260806 16:16:08.047274 4332 leader_election.cc:302] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5991b0e2c5414799866035d84d154cb3, a467d4bdc7114ea3ac40cfc0386a171e; no voters:
I20260806 16:16:08.047457 4603 raft_consensus.cc:2781] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:08.047711 4603 raft_consensus.cc:686] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 5991b0e2c5414799866035d84d154cb3, State: Running, Role: LEADER
I20260806 16:16:08.048069 4603 consensus_queue.cc:237] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } }
I20260806 16:16:08.051520 4122 catalog_manager.cc:5495] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5991b0e2c5414799866035d84d154cb3 (127.18.90.2). New cstate: current_term: 1 leader_uuid: "5991b0e2c5414799866035d84d154cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5991b0e2c5414799866035d84d154cb3" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44107 } health_report { overall_health: HEALTHY } } }
I20260806 16:16:08.087014 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:08.083669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.18.90.2:44107)
0806 16:16:08.083819 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:08.083822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:08.083835 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:16:08.083913 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:08.083915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:08.083969 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:08.084058 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:08.084371 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.084858 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.085672 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:08.085883 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:08.085959 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:08.086251 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:08.086679 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:08.086780 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:08.086837 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 16:16:08.086884 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 16:16:08.093353 4266 raft_consensus.cc:1261] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e [term 1 FOLLOWER]: Refusing update from remote peer 5991b0e2c5414799866035d84d154cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:08.093371 4531 raft_consensus.cc:1261] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 1 FOLLOWER]: Refusing update from remote peer 5991b0e2c5414799866035d84d154cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:08.094175 4603 consensus_queue.cc:1035] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f2942ee6024d59983801b398aac016" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:08.094496 4606 consensus_queue.cc:1035] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a467d4bdc7114ea3ac40cfc0386a171e" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:08.104382 4610 mvcc.cc:205] Tried to move back new op lower bound from 7315591037302030336 to 7315591037128536064. Current Snapshot: MvccSnapshot[applied={T|T < 7315591037302030336}]
I20260806 16:16:08.106492 4611 mvcc.cc:205] Tried to move back new op lower bound from 7315591037302030336 to 7315591037128536064. Current Snapshot: MvccSnapshot[applied={T|T < 7315591037302030336 or (T in {7315591037302030336})}]
I20260806 16:16:08.110739 4122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:08.110927 4122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:08.113525 4122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e04947d97144ee4aa265979c78eef18: Sending DeleteTablet for 3 replicas of tablet 1f9bf840b39b45deb571ff64ea6e32f9
I20260806 16:16:08.114373 4246 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9bf840b39b45deb571ff64ea6e32f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:08 UTC) from {username='slave'} at 127.0.0.1:57304
I20260806 16:16:08.114516 4511 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9bf840b39b45deb571ff64ea6e32f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:08 UTC) from {username='slave'} at 127.0.0.1:51866
I20260806 16:16:08.114853 4379 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9bf840b39b45deb571ff64ea6e32f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:08 UTC) from {username='slave'} at 127.0.0.1:59880
I20260806 16:16:08.114943 4618 tablet_replica.cc:330] T 1f9bf840b39b45deb571ff64ea6e32f9 P a467d4bdc7114ea3ac40cfc0386a171e: stopping tablet replica
I20260806 16:16:08.115152 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4183
I20260806 16:16:08.117097 4619 tablet_replica.cc:330] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: stopping tablet replica
I20260806 16:16:08.117446 4619 raft_consensus.cc:2229] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:08.117776 4619 raft_consensus.cc:2258] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:08.118257 4620 tablet_replica.cc:330] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: stopping tablet replica
I20260806 16:16:08.118527 4620 raft_consensus.cc:2229] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:16:08.118889 4620 raft_consensus.cc:2258] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:08.120427 4619 ts_tablet_manager.cc:1905] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:08.120427 4620 ts_tablet_manager.cc:1905] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:08.123274 4619 ts_tablet_manager.cc:1918] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:08.123287 4620 ts_tablet_manager.cc:1918] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:08.123392 4619 log.cc:1196] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/1f9bf840b39b45deb571ff64ea6e32f9
I20260806 16:16:08.123513 4620 log.cc:1196] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/1f9bf840b39b45deb571ff64ea6e32f9
I20260806 16:16:08.123745 4619 ts_tablet_manager.cc:1939] T 1f9bf840b39b45deb571ff64ea6e32f9 P 88f2942ee6024d59983801b398aac016: Deleting consensus metadata
I20260806 16:16:08.123977 4620 ts_tablet_manager.cc:1939] T 1f9bf840b39b45deb571ff64ea6e32f9 P 5991b0e2c5414799866035d84d154cb3: Deleting consensus metadata
W20260806 16:16:08.124660 4105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:38923 (error 108)
I20260806 16:16:08.125198 4105 catalog_manager.cc:4841] TS 5991b0e2c5414799866035d84d154cb3 (127.18.90.2:44107): tablet 1f9bf840b39b45deb571ff64ea6e32f9 (table test-table [id=e4b55b3e14de47d8b4e4fd5ef8e880aa]) successfully deleted
I20260806 16:16:08.125203 4108 catalog_manager.cc:4841] TS 88f2942ee6024d59983801b398aac016 (127.18.90.3:45527): tablet 1f9bf840b39b45deb571ff64ea6e32f9 (table test-table [id=e4b55b3e14de47d8b4e4fd5ef8e880aa]) successfully deleted
I20260806 16:16:08.125898 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4316
I20260806 16:16:08.126289 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:08.125927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:38923 (local address 127.0.0.1:57310)
0806 16:16:08.126104 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:08.126199 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 16:16:08.126626 4105 catalog_manager.cc:4584] TS a467d4bdc7114ea3ac40cfc0386a171e (127.18.90.1:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f9bf840b39b45deb571ff64ea6e32f9: Network error: Client connection negotiation failed: client connection to 127.18.90.1:38923: connect: Connection refused (error 111)
I20260806 16:16:08.132221 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4449
I20260806 16:16:08.138041 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4090
2026-08-06T16:16:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1079 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:08 dist-test-slave-0b58 krb5kdc[4624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:08 dist-test-slave-0b58 krb5kdc[4624](info): set up 2 sockets
Aug 06 16:16:08 dist-test-slave-0b58 krb5kdc[4624](info): commencing operation
krb5kdc: starting...
W20260806 16:16:09.172526 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:09 dist-test-slave-0b58 krb5kdc[4624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:09.293566 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:34997
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:09.403189 4637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:09.403442 4637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:09.403482 4637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:09.407451 4637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:09.407506 4637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:09.407526 4637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:09.407541 4637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:09.407598 4637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:09.412819 4637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:34997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:34997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:09.413894 4637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:09.414907 4637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:09.423592 4643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:09.423552 4642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:09.423633 4637 server_base.cc:1034] running on GCE node
W20260806 16:16:09.423547 4645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:09.424540 4637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:09.425715 4637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:09.426913 4637 hybrid_clock.cc:648] HybridClock initialized: now 1786032969426897 us; error 26 us; skew 500 ppm
Aug 06 16:16:09 dist-test-slave-0b58 krb5kdc[4624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:09.429774 4637 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:09.430723 4637 webserver.cc:460] Webserver started at http://127.18.90.62:33821/ using document root <none> and password file <none>
I20260806 16:16:09.431360 4637 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:09.431455 4637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:09.431681 4637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:09.433833 4637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "17a7eeb69a27410dbb444a63dc3ed870"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "98fb1d3f838311c70710703f87be357e"
server_key_iv: "0130b6a001d9d0cc5d1c6583fa42073f"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.434418 4637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "17a7eeb69a27410dbb444a63dc3ed870"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "98fb1d3f838311c70710703f87be357e"
server_key_iv: "0130b6a001d9d0cc5d1c6583fa42073f"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.438321 4637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:16:09.440883 4653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.442240 4637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:16:09.442401 4637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "17a7eeb69a27410dbb444a63dc3ed870"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "98fb1d3f838311c70710703f87be357e"
server_key_iv: "0130b6a001d9d0cc5d1c6583fa42073f"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.442525 4637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:09.455123 4637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:09.458055 4637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:09.458274 4637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:09.465341 4637 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:34997
I20260806 16:16:09.465376 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:34997 every 8 connection(s)
I20260806 16:16:09.466307 4637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:09.468979 4706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:09.469942 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4637
I20260806 16:16:09.470090 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:09.470439 18792 external_mini_cluster.cc:1428] Setting key b2d13715a9a93bed2d3a5a15ad941f54
I20260806 16:16:09.475194 4706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870: Bootstrap starting.
I20260806 16:16:09.478575 4706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870: Neither blocks nor log segments found. Creating new log.
Aug 06 16:16:09 dist-test-slave-0b58 krb5kdc[4624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:09.479825 4706 log.cc:824] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:09.482846 4706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870: No bootstrap required, opened a new log
I20260806 16:16:09.485071 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:09.471893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.18.90.62:34997)
0806 16:16:09.472399 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:09.472415 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:09.472451 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:16:09.473106 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:09.473124 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:09.473459 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:09.473816 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:09.474682 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:09.475708 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:09.476751 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:09.477241 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:09.477427 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:09.480121 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:09.480126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:09.480140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:09.480176 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:09.482002 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:09.482468 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:09.482474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:09.482479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:09.482591 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:09.482944 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:09.482947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:09.482949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:09.483412 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:09.483623 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:09.483921 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 16:16:09.484231 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260806 16:16:09.487315 4706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7eeb69a27410dbb444a63dc3ed870" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34997 } }
I20260806 16:16:09.487634 4706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:09.487694 4706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a7eeb69a27410dbb444a63dc3ed870, State: Initialized, Role: FOLLOWER
I20260806 16:16:09.488097 4706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7eeb69a27410dbb444a63dc3ed870" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34997 } }
I20260806 16:16:09.488214 4706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:09.488276 4706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:09.488564 4706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:09.489673 4706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7eeb69a27410dbb444a63dc3ed870" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34997 } }
I20260806 16:16:09.490028 4706 leader_election.cc:302] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a7eeb69a27410dbb444a63dc3ed870; no voters:
I20260806 16:16:09.490424 4706 leader_election.cc:288] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:09.490509 4711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:09.490692 4711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [term 1 LEADER]: Becoming Leader. State: Replica: 17a7eeb69a27410dbb444a63dc3ed870, State: Running, Role: LEADER
I20260806 16:16:09.490990 4711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7eeb69a27410dbb444a63dc3ed870" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34997 } }
I20260806 16:16:09.491786 4706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:09.492799 4712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a7eeb69a27410dbb444a63dc3ed870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7eeb69a27410dbb444a63dc3ed870" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34997 } } }
I20260806 16:16:09.492944 4712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:09.492903 4713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a7eeb69a27410dbb444a63dc3ed870. Latest consensus state: current_term: 1 leader_uuid: "17a7eeb69a27410dbb444a63dc3ed870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7eeb69a27410dbb444a63dc3ed870" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 34997 } } }
I20260806 16:16:09.493014 4713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:09.493350 4720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:09.496469 4720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:09.502116 4720 catalog_manager.cc:1347] Generated new cluster ID: 584f57aabad0442cba7ae5302e02635f
I20260806 16:16:09.502192 4720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:09.510310 4720 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:09.511559 4720 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:09.523643 4720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870: Generated new TSK 0
I20260806 16:16:09.524510 4720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:16:09.559297 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34997
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:09.674561 4732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:09.674867 4732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:09.674949 4732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:09.679118 4732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:09.679221 4732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:09.679323 4732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:16:09.684737 4732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:09.685951 4732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:09.687060 4732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:09.697026 4738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:09.697072 4737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:09.697156 4732 server_base.cc:1034] running on GCE node
W20260806 16:16:09.697052 4740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:09.697780 4732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:09.698441 4732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:09.699640 4732 hybrid_clock.cc:648] HybridClock initialized: now 1786032969699618 us; error 32 us; skew 500 ppm
Aug 06 16:16:09 dist-test-slave-0b58 krb5kdc[4624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:09.702406 4732 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:16:09.703297 4732 webserver.cc:460] Webserver started at http://127.18.90.1:39783/ using document root <none> and password file <none>
I20260806 16:16:09.703882 4732 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:09.703977 4732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:09.704214 4732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:09.706457 4732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "0f63d08ae2ca473eb83204d9e16ffcb5"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "2249adf2d982944fec474e5be8c3e90b"
server_key_iv: "f5c533d396c4e0b129ac895dd2dd7545"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.707046 4732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "0f63d08ae2ca473eb83204d9e16ffcb5"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "2249adf2d982944fec474e5be8c3e90b"
server_key_iv: "f5c533d396c4e0b129ac895dd2dd7545"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.710988 4732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:16:09.713501 4747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.714751 4732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:09.714887 4732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "0f63d08ae2ca473eb83204d9e16ffcb5"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "2249adf2d982944fec474e5be8c3e90b"
server_key_iv: "f5c533d396c4e0b129ac895dd2dd7545"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.715003 4732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:09.733585 4732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:09.736820 4732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:09.737041 4732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:09.737694 4732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:09.738577 4732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:09.738651 4732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.738729 4732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:09.738782 4732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.748517 4732 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:34231
I20260806 16:16:09.748534 4862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:34231 every 8 connection(s)
I20260806 16:16:09.749495 4732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:16:09.756840 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4732
I20260806 16:16:09.757005 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:16:09.757321 18792 external_mini_cluster.cc:1428] Setting key 086387d8f3a8be65c66d6471c2e9c321
Aug 06 16:16:09 dist-test-slave-0b58 krb5kdc[4624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:09.764220 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:09.751364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:43423 (local address 127.18.90.62:34997)
0806 16:16:09.751540 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:09.751545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:09.752464 (+ 919us) server_negotiation.cc:407] Connection header received
0806 16:16:09.753786 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:09.753789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:09.753837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:09.753911 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:09.755067 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:09.755608 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:09.757001 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:09.757314 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:09.757449 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:09.760167 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:09.760173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:09.760179 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:09.760212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:09.761773 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:09.762455 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:09.762458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:09.762459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:09.762531 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:09.762918 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:09.762924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:09.762928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:09.763193 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:09.763328 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:09.763793 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 16:16:09.763947 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 16:16:09.764951 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:09.751778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34997 (local address 127.18.90.1:43423)
0806 16:16:09.752298 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:09.752360 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:09.753587 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:09.754079 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:09.754085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:09.754440 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:09.754932 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:09.754946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:09.755716 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:09.755718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:09.756850 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:09.756863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:09.757507 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:09.757511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:09.757705 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:09.758336 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:09.758356 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:09.759990 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:09.761903 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:09.761910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:09.761927 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:09.762344 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:09.762667 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:09.762672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:09.762674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:09.762818 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:09.763327 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:09.763331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:09.763552 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 16:16:09.763706 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 16:16:09.763956 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
I20260806 16:16:09.766258 4863 heartbeater.cc:347] Connected to a master server at 127.18.90.62:34997
I20260806 16:16:09.766522 4863 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:09.767155 4863 heartbeater.cc:511] Master 127.18.90.62:34997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:16:09.768855 4670 ts_manager.cc:194] Registered new tserver with Master: 0f63d08ae2ca473eb83204d9e16ffcb5 (127.18.90.1:34231)
I20260806 16:16:09.770149 4670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:43423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:16:09.787649 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34997
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:09.900662 4868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:09.900921 4868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:09.900964 4868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:09.905019 4868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:09.905072 4868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:09.905129 4868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:16:09.910336 4868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:09.911329 4868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:09.912336 4868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:09.922250 4874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:09.922230 4876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:09.922283 4873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:09.922420 4868 server_base.cc:1034] running on GCE node
I20260806 16:16:09.922894 4868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:09.923525 4868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:09.924738 4868 hybrid_clock.cc:648] HybridClock initialized: now 1786032969924722 us; error 28 us; skew 500 ppm
Aug 06 16:16:09 dist-test-slave-0b58 krb5kdc[4624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:09.931860 4868 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:16:09.932878 4868 webserver.cc:460] Webserver started at http://127.18.90.2:37597/ using document root <none> and password file <none>
I20260806 16:16:09.933437 4868 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:09.933501 4868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:09.933673 4868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:09.935828 4868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "73857a39256c4f7da438e55c6254a24d"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "8ce2d06e8ca8ccb4adcc4d2c211b9a16"
server_key_iv: "cd2d2051903ea4db05b4353259b27cc3"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.936496 4868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "73857a39256c4f7da438e55c6254a24d"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "8ce2d06e8ca8ccb4adcc4d2c211b9a16"
server_key_iv: "cd2d2051903ea4db05b4353259b27cc3"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.940372 4868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:16:09.942889 4883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.944123 4868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:09.944255 4868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "73857a39256c4f7da438e55c6254a24d"
format_stamp: "Formatted at 2026-08-06 16:16:09 on dist-test-slave-0b58"
server_key: "8ce2d06e8ca8ccb4adcc4d2c211b9a16"
server_key_iv: "cd2d2051903ea4db05b4353259b27cc3"
server_key_version: "encryptionkey@0"
I20260806 16:16:09.944392 4868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:09.983237 4868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:09.986848 4868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:09.987062 4868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:09.987694 4868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:09.988602 4868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:09.988651 4868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.988694 4868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:09.988749 4868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:09.998373 4868 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:46497
I20260806 16:16:09.998401 4997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:46497 every 8 connection(s)
I20260806 16:16:09.999323 4868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:16:10.004964 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 4868
I20260806 16:16:10.005098 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:16:10.005445 18792 external_mini_cluster.cc:1428] Setting key a6c8fa44a682e69e87e667060b31b03c
Aug 06 16:16:10 dist-test-slave-0b58 krb5kdc[4624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032969, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:10.014684 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.001267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:43457 (local address 127.18.90.62:34997)
0806 16:16:10.001408 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.001411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.002318 (+ 907us) server_negotiation.cc:407] Connection header received
0806 16:16:10.003728 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.003731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.003791 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.003908 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:10.005310 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.006042 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.007365 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.007739 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.007871 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.010854 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:10.010857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:10.010859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:10.010884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:10.012516 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:10.013061 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:10.013065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:10.013067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:10.013153 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:10.013515 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:10.013518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:10.013520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:10.013709 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:10.013815 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.014369 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.014508 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
I20260806 16:16:10.015573 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.001616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34997 (local address 127.18.90.2:43457)
0806 16:16:10.002146 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.002192 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.003501 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.004094 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.004101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.004557 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:10.005064 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.005106 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.006190 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.006192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.007231 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.007241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.007894 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.007897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.008044 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.008902 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:10.008928 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:10.010668 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:10.012649 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:10.012662 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:10.012676 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:10.012948 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:10.013250 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:10.013256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:10.013260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:10.013412 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:10.013829 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:10.013837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:10.014123 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.014326 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.014661 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":131,"threads_started":1}
I20260806 16:16:10.016920 4998 heartbeater.cc:347] Connected to a master server at 127.18.90.62:34997
I20260806 16:16:10.017146 4998 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:10.017688 4998 heartbeater.cc:511] Master 127.18.90.62:34997 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
I20260806 16:16:10.019043 4670 ts_manager.cc:194] Registered new tserver with Master: 73857a39256c4f7da438e55c6254a24d (127.18.90.2:46497)
I20260806 16:16:10.019721 4670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:43457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:16:10.034853 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34997
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:10.147060 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:10.147317 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:10.147359 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:10.151356 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:10.151412 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:10.151464 5003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:16:10.156440 5003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:10.157382 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:10.158311 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:10.167615 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:10.167615 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:10.167616 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:10.167865 5003 server_base.cc:1034] running on GCE node
I20260806 16:16:10.168442 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:10.169010 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:10.170174 5003 hybrid_clock.cc:648] HybridClock initialized: now 1786032970170157 us; error 30 us; skew 500 ppm
Aug 06 16:16:10 dist-test-slave-0b58 krb5kdc[4624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:10.172916 5003 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:16:10.173692 5003 webserver.cc:460] Webserver started at http://127.18.90.3:43973/ using document root <none> and password file <none>
I20260806 16:16:10.174201 5003 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:10.174266 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:10.174427 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:10.176436 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "f4996ca7f64e410c98f60a390838b181"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "980a9aff337cbf6405abec82a2c55a05"
server_key_iv: "0502beda051577fcc56426137cee8d68"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.176963 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "f4996ca7f64e410c98f60a390838b181"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "980a9aff337cbf6405abec82a2c55a05"
server_key_iv: "0502beda051577fcc56426137cee8d68"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.180527 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:10.182785 5018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:10.183928 5003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:10.184064 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "f4996ca7f64e410c98f60a390838b181"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "980a9aff337cbf6405abec82a2c55a05"
server_key_iv: "0502beda051577fcc56426137cee8d68"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.184172 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:10.216012 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:10.219182 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:10.219421 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:10.220070 5003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:10.220912 5003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:10.220983 5003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:10.221048 5003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:10.221077 5003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:10.232463 5003 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:37713
I20260806 16:16:10.232486 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:37713 every 8 connection(s)
I20260806 16:16:10.233415 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:16:10.242029 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5003
I20260806 16:16:10.242192 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:16:10.242506 18792 external_mini_cluster.cc:1428] Setting key b220b0d51956954e2f81c6a888ef702f
Aug 06 16:16:10 dist-test-slave-0b58 krb5kdc[4624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:10.247570 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.235307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:36783 (local address 127.18.90.62:34997)
0806 16:16:10.235452 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.235456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.236318 (+ 862us) server_negotiation.cc:407] Connection header received
0806 16:16:10.237691 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.237695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.237753 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.237871 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:10.239028 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.239569 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.240866 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.241101 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.241219 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.244237 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:10.244240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:10.244242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:10.244299 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:10.245727 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:10.246225 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:10.246227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:10.246229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:10.246303 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:10.246632 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:10.246634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:10.246635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:10.246810 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:10.246900 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.247353 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.247452 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:16:10.248620 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.235714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:34997 (local address 127.18.90.3:36783)
0806 16:16:10.236146 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.236194 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.237480 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.238033 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.238040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.238397 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:10.238883 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.238897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.239712 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.239717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.240739 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.240750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.241259 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.241263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.241431 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.242158 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:10.242185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:10.244075 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:10.245831 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:10.245836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:10.245848 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:10.246140 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:10.246389 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:10.246391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:10.246393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:10.246545 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:10.246912 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:10.246917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:10.247150 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.247307 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.247561 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260806 16:16:10.249727 5135 heartbeater.cc:347] Connected to a master server at 127.18.90.62:34997
I20260806 16:16:10.249970 5135 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:10.250543 5135 heartbeater.cc:511] Master 127.18.90.62:34997 requested a full tablet report, sending...
I20260806 16:16:10.251495 4670 ts_manager.cc:194] Registered new tserver with Master: f4996ca7f64e410c98f60a390838b181 (127.18.90.3:37713)
I20260806 16:16:10.252053 4670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:36783
I20260806 16:16:10.256430 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:16:10.266234 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.258459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.18.90.62:34997)
0806 16:16:10.258578 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.258582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.258850 (+ 268us) server_negotiation.cc:407] Connection header received
0806 16:16:10.258947 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.258949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.258987 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.259059 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:10.259798 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.260367 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.261244 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.261460 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.261539 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.263132 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:10.263136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:10.263138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:10.263160 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:10.264647 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:10.265106 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:10.265109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:10.265110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:10.265181 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:10.265496 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:10.265499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:10.265500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:10.265686 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:10.265803 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.266029 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.266105 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:16:10.269176 4670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:16:10.272138 4670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:16:10.286417 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.281066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:37713 (local address 127.0.0.1:39092)
0806 16:16:10.281771 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.281783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.281854 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.282426 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.282431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.282453 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:10.282556 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.282561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.283793 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.283799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.284950 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.284963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.285805 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.285809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.285927 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.285966 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.286103 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.286241 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":58,"thread_start_us":93,"threads_started":1}
I20260806 16:16:10.286656 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.280153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:46497 (local address 127.0.0.1:42672)
0806 16:16:10.280963 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.281027 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.281162 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.281759 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.281765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.281791 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:10.281954 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.281961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.283396 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.283400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.284854 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.284865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.286169 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.286173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.286331 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.286354 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.286425 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.286490 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":28,"thread_start_us":75,"threads_started":1}
I20260806 16:16:10.286809 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.280236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34231 (local address 127.0.0.1:37058)
0806 16:16:10.281499 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.281514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.281593 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.282477 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.282479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.282491 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:10.282555 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.282561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.283881 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.283884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.285054 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.285065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.286092 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.286099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.286222 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.286242 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.286331 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.286385 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260806 16:16:10.286914 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.281478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.18.90.3:37713)
0806 16:16:10.281960 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.281972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.281996 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:16:10.282084 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.282088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.282278 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.282428 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:10.282669 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.283668 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.285082 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.285674 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.285804 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.285959 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.286634 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.286736 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":98,"threads_started":1}
I20260806 16:16:10.287170 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.280310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.18.90.2:46497)
0806 16:16:10.280685 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.280698 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.281307 (+ 609us) server_negotiation.cc:407] Connection header received
0806 16:16:10.281401 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.281407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.281611 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.281790 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:10.282066 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.283250 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.284988 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.285829 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.285941 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.286086 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.286871 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.286995 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":154,"threads_started":1}
I20260806 16:16:10.287406 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.280963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.18.90.1:34231)
0806 16:16:10.282101 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.282106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.282124 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:16:10.282182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.282185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.282325 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.282703 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:10.282825 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.283681 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.285175 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.285973 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.286079 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.286233 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.286366 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.286474 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":112,"threads_started":1}
I20260806 16:16:10.289981 4797 tablet_service.cc:1467] Processing CreateTablet for tablet 1a223d92e14e47b8a90fb0914316868c (DEFAULT_TABLE table=test-table [id=1bef1d9c3fa5473e8fb3594603ee68da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:10.290002 5068 tablet_service.cc:1467] Processing CreateTablet for tablet 1a223d92e14e47b8a90fb0914316868c (DEFAULT_TABLE table=test-table [id=1bef1d9c3fa5473e8fb3594603ee68da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:10.290005 4932 tablet_service.cc:1467] Processing CreateTablet for tablet 1a223d92e14e47b8a90fb0914316868c (DEFAULT_TABLE table=test-table [id=1bef1d9c3fa5473e8fb3594603ee68da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:10.290971 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a223d92e14e47b8a90fb0914316868c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:10.291021 5068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a223d92e14e47b8a90fb0914316868c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:10.291013 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a223d92e14e47b8a90fb0914316868c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:10.298425 5154 tablet_bootstrap.cc:492] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Bootstrap starting.
I20260806 16:16:10.298897 5153 tablet_bootstrap.cc:492] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: Bootstrap starting.
I20260806 16:16:10.300547 5154 tablet_bootstrap.cc:654] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:10.300628 5155 tablet_bootstrap.cc:492] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Bootstrap starting.
I20260806 16:16:10.300666 5153 tablet_bootstrap.cc:654] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:10.301364 5153 log.cc:824] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:10.301879 5154 log.cc:824] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:10.302799 5155 tablet_bootstrap.cc:654] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:10.303570 5153 tablet_bootstrap.cc:492] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: No bootstrap required, opened a new log
I20260806 16:16:10.303725 5154 tablet_bootstrap.cc:492] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: No bootstrap required, opened a new log
I20260806 16:16:10.303790 5153 ts_tablet_manager.cc:1397] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 16:16:10.303886 5154 ts_tablet_manager.cc:1397] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:10.304180 5155 log.cc:824] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:10.305795 5155 tablet_bootstrap.cc:492] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: No bootstrap required, opened a new log
I20260806 16:16:10.305994 5155 ts_tablet_manager.cc:1397] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:10.307963 5154 raft_consensus.cc:348] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.308303 5154 raft_consensus.cc:374] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:10.308408 5154 raft_consensus.cc:729] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4996ca7f64e410c98f60a390838b181, State: Initialized, Role: FOLLOWER
I20260806 16:16:10.308377 5153 raft_consensus.cc:348] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.308633 5153 raft_consensus.cc:374] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:10.308725 5153 raft_consensus.cc:729] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f63d08ae2ca473eb83204d9e16ffcb5, State: Initialized, Role: FOLLOWER
I20260806 16:16:10.308966 5154 consensus_queue.cc:260] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.309142 5153 consensus_queue.cc:260] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.309003 5155 raft_consensus.cc:348] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.309319 5155 raft_consensus.cc:374] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:10.309422 5155 raft_consensus.cc:729] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73857a39256c4f7da438e55c6254a24d, State: Initialized, Role: FOLLOWER
I20260806 16:16:10.309888 5155 consensus_queue.cc:260] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.310073 5153 ts_tablet_manager.cc:1428] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 16:16:10.310089 5154 ts_tablet_manager.cc:1428] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:16:10.310509 4863 heartbeater.cc:503] Master 127.18.90.62:34997 was elected leader, sending a full tablet report...
I20260806 16:16:10.310798 4998 heartbeater.cc:503] Master 127.18.90.62:34997 was elected leader, sending a full tablet report...
I20260806 16:16:10.310822 5135 heartbeater.cc:503] Master 127.18.90.62:34997 was elected leader, sending a full tablet report...
I20260806 16:16:10.311846 5155 ts_tablet_manager.cc:1428] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:16:10.466666 5161 raft_consensus.cc:482] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:10.466903 5161 raft_consensus.cc:504] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.469085 5161 leader_election.cc:288] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f63d08ae2ca473eb83204d9e16ffcb5 (127.18.90.1:34231), f4996ca7f64e410c98f60a390838b181 (127.18.90.3:37713)
I20260806 16:16:10.473150 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.468448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34231 (local address 127.18.90.2:42965)
0806 16:16:10.468750 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.468780 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.468866 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.469551 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.469553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.469571 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:10.469734 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.469742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.470786 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.470790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.471897 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.471906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.472696 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.472699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.472815 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.472837 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.472910 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.472990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":27}
I20260806 16:16:10.473160 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.469159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:42965 (local address 127.18.90.1:34231)
0806 16:16:10.469310 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.469314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.469328 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:16:10.469384 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.469387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.469449 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.469535 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:10.469876 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.470666 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.472066 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.472575 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.472695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.472760 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.472943 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.473039 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:16:10.473287 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.468857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:52859 (local address 127.18.90.3:37713)
0806 16:16:10.469012 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.469016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.469241 (+ 225us) server_negotiation.cc:407] Connection header received
0806 16:16:10.469449 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.469453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.469512 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.469596 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:10.469910 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.470438 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.472036 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.472695 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.472809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.472869 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.473079 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.473154 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 16:16:10.473681 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.468470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:37713 (local address 127.18.90.2:52859)
0806 16:16:10.469154 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.469167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:10.469314 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:10.469659 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:10.469664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:10.469678 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:10.469810 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.469818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.470556 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.470561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.471895 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:10.471906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.472832 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.472835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:10.472936 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.472954 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:16:10.473443 (+ 489us) client_negotiation.cc:240] Negotiation successful
0806 16:16:10.473542 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":8,"thread_start_us":75,"threads_started":1}
I20260806 16:16:10.473892 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a223d92e14e47b8a90fb0914316868c" candidate_uuid: "73857a39256c4f7da438e55c6254a24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" is_pre_election: true
I20260806 16:16:10.474236 4817 raft_consensus.cc:2454] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73857a39256c4f7da438e55c6254a24d in term 0.
I20260806 16:16:10.474395 5085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a223d92e14e47b8a90fb0914316868c" candidate_uuid: "73857a39256c4f7da438e55c6254a24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4996ca7f64e410c98f60a390838b181" is_pre_election: true
I20260806 16:16:10.474728 5085 raft_consensus.cc:2454] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73857a39256c4f7da438e55c6254a24d in term 0.
I20260806 16:16:10.474826 4887 leader_election.cc:302] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f63d08ae2ca473eb83204d9e16ffcb5, 73857a39256c4f7da438e55c6254a24d; no voters:
I20260806 16:16:10.475126 5161 raft_consensus.cc:2781] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:16:10.475222 5161 raft_consensus.cc:482] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:16:10.475265 5161 raft_consensus.cc:3037] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:10.476558 5161 raft_consensus.cc:504] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.476936 5161 leader_election.cc:288] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [CANDIDATE]: Term 1 election: Requested vote from peers 0f63d08ae2ca473eb83204d9e16ffcb5 (127.18.90.1:34231), f4996ca7f64e410c98f60a390838b181 (127.18.90.3:37713)
I20260806 16:16:10.477285 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a223d92e14e47b8a90fb0914316868c" candidate_uuid: "73857a39256c4f7da438e55c6254a24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5"
I20260806 16:16:10.477360 5085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a223d92e14e47b8a90fb0914316868c" candidate_uuid: "73857a39256c4f7da438e55c6254a24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4996ca7f64e410c98f60a390838b181"
I20260806 16:16:10.477427 4817 raft_consensus.cc:3037] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:10.477483 5085 raft_consensus.cc:3037] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:10.478466 4817 raft_consensus.cc:2454] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73857a39256c4f7da438e55c6254a24d in term 1.
I20260806 16:16:10.478768 5085 raft_consensus.cc:2454] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73857a39256c4f7da438e55c6254a24d in term 1.
I20260806 16:16:10.478789 4887 leader_election.cc:302] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f63d08ae2ca473eb83204d9e16ffcb5, 73857a39256c4f7da438e55c6254a24d; no voters:
I20260806 16:16:10.478991 5161 raft_consensus.cc:2781] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:10.479212 5161 raft_consensus.cc:686] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 1 LEADER]: Becoming Leader. State: Replica: 73857a39256c4f7da438e55c6254a24d, State: Running, Role: LEADER
I20260806 16:16:10.479552 5161 consensus_queue.cc:237] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } }
I20260806 16:16:10.482893 4670 catalog_manager.cc:5495] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73857a39256c4f7da438e55c6254a24d (127.18.90.2). New cstate: current_term: 1 leader_uuid: "73857a39256c4f7da438e55c6254a24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73857a39256c4f7da438e55c6254a24d" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 46497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 } health_report { overall_health: UNKNOWN } } }
W20260806 16:16:10.484429 5136 tablet.cc:2369] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:16:10.500444 4999 tablet.cc:2369] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:16:10.500841 4864 tablet.cc:2369] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:16:10.523195 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.519328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.18.90.2:46497)
0806 16:16:10.519465 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.519468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.519500 (+ 32us) server_negotiation.cc:407] Connection header received
0806 16:16:10.519728 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.519732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.519809 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.519927 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:10.520182 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.520772 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.521635 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.521870 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.521979 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.522312 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:10.522735 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:10.522897 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.522984 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.523022 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:10.529121 5085 raft_consensus.cc:1261] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 1 FOLLOWER]: Refusing update from remote peer 73857a39256c4f7da438e55c6254a24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:10.529121 4817 raft_consensus.cc:1261] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 1 FOLLOWER]: Refusing update from remote peer 73857a39256c4f7da438e55c6254a24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:10.529907 5163 consensus_queue.cc:1035] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4996ca7f64e410c98f60a390838b181" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:10.530095 5161 consensus_queue.cc:1035] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f63d08ae2ca473eb83204d9e16ffcb5" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:10.539528 5167 mvcc.cc:205] Tried to move back new op lower bound from 7315591047279718400 to 7315591047087325184. Current Snapshot: MvccSnapshot[applied={T|T < 7315591047279718400}]
I20260806 16:16:10.547695 4670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:10.547904 4670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:10.550586 4670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17a7eeb69a27410dbb444a63dc3ed870: Sending DeleteTablet for 3 replicas of tablet 1a223d92e14e47b8a90fb0914316868c
I20260806 16:16:10.551488 4797 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a223d92e14e47b8a90fb0914316868c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:10 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:37058
I20260806 16:16:10.551607 5068 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a223d92e14e47b8a90fb0914316868c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:10 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:39092
I20260806 16:16:10.551882 4932 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a223d92e14e47b8a90fb0914316868c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:10 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:42672
I20260806 16:16:10.552052 5177 tablet_replica.cc:330] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5: stopping tablet replica
I20260806 16:16:10.552464 5176 tablet_replica.cc:330] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: stopping tablet replica
I20260806 16:16:10.552518 5178 tablet_replica.cc:330] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: stopping tablet replica
I20260806 16:16:10.552838 5178 raft_consensus.cc:2229] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:16:10.552959 5177 raft_consensus.cc:2229] T 1a223d92e14e47b8a90fb0914316868c P 0f63d08ae2ca473eb83204d9e16ffcb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:10.553082 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4732
I20260806 16:16:10.554332 5178 raft_consensus.cc:2258] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:10.555960 5178 ts_tablet_manager.cc:1905] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:10.556111 5176 raft_consensus.cc:2229] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:10.556483 5176 raft_consensus.cc:2258] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:10.558132 5176 ts_tablet_manager.cc:1905] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:10.558992 5178 ts_tablet_manager.cc:1918] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:10.559135 5178 log.cc:1196] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/1a223d92e14e47b8a90fb0914316868c
I20260806 16:16:10.559556 5178 ts_tablet_manager.cc:1939] T 1a223d92e14e47b8a90fb0914316868c P 73857a39256c4f7da438e55c6254a24d: Deleting consensus metadata
I20260806 16:16:10.560881 4655 catalog_manager.cc:4841] TS 73857a39256c4f7da438e55c6254a24d (127.18.90.2:46497): tablet 1a223d92e14e47b8a90fb0914316868c (table test-table [id=1bef1d9c3fa5473e8fb3594603ee68da]) successfully deleted
I20260806 16:16:10.560940 5176 ts_tablet_manager.cc:1918] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:10.561019 5176 log.cc:1196] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/1a223d92e14e47b8a90fb0914316868c
I20260806 16:16:10.561329 5176 ts_tablet_manager.cc:1939] T 1a223d92e14e47b8a90fb0914316868c P f4996ca7f64e410c98f60a390838b181: Deleting consensus metadata
W20260806 16:16:10.562028 4657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:34231 (error 108)
I20260806 16:16:10.562541 4654 catalog_manager.cc:4841] TS f4996ca7f64e410c98f60a390838b181 (127.18.90.3:37713): tablet 1a223d92e14e47b8a90fb0914316868c (table test-table [id=1bef1d9c3fa5473e8fb3594603ee68da]) successfully deleted
I20260806 16:16:10.562830 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4868
I20260806 16:16:10.563015 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.562746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:34231 (local address 127.0.0.1:37066)
0806 16:16:10.562854 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:10.562925 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 16:16:10.564983 4657 catalog_manager.cc:4584] TS 0f63d08ae2ca473eb83204d9e16ffcb5 (127.18.90.1:34231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a223d92e14e47b8a90fb0914316868c: Network error: Client connection negotiation failed: client connection to 127.18.90.1:34231: connect: Connection refused (error 111)
I20260806 16:16:10.569521 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5003
I20260806 16:16:10.575150 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 4637
2026-08-06T16:16:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2436 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T16:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:10Z Disabled control of system clock
I20260806 16:16:10.607290 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36717
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:36717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:16:10.718786 5183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:10.719056 5183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:10.719091 5183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:10.723235 5183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:10.723295 5183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:10.723311 5183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:10.723327 5183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:10.723342 5183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:10.727984 5183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:36717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:10.729084 5183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:10.730415 5183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:10.738852 5191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:10.738925 5188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:10.739133 5183 server_base.cc:1034] running on GCE node
W20260806 16:16:10.738933 5189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:10.739694 5183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:10.740865 5183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:10.742026 5183 hybrid_clock.cc:648] HybridClock initialized: now 1786032970741995 us; error 44 us; skew 500 ppm
I20260806 16:16:10.743909 5183 webserver.cc:460] Webserver started at http://127.18.90.62:33933/ using document root <none> and password file <none>
I20260806 16:16:10.744563 5183 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:10.744628 5183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:10.744892 5183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:10.746980 5183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "e3a1a391c6084bc88ca42f22573d2191"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "ab2f96ac1fa1106e1b44df9591ce75f0"
server_key_iv: "fa88b7b7efbe923dcafca768adca1ecd"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.747570 5183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "e3a1a391c6084bc88ca42f22573d2191"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "ab2f96ac1fa1106e1b44df9591ce75f0"
server_key_iv: "fa88b7b7efbe923dcafca768adca1ecd"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.751235 5183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:16:10.753679 5197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:10.754860 5183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:10.755002 5183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "e3a1a391c6084bc88ca42f22573d2191"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "ab2f96ac1fa1106e1b44df9591ce75f0"
server_key_iv: "fa88b7b7efbe923dcafca768adca1ecd"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.755126 5183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:10.774299 5183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:10.774991 5183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:10.775202 5183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:10.782467 5183 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:36717
I20260806 16:16:10.782470 5249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:36717 every 8 connection(s)
I20260806 16:16:10.783425 5183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:10.786582 5250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:10.792109 5250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191: Bootstrap starting.
I20260806 16:16:10.793529 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5183
I20260806 16:16:10.793680 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:10.793956 18792 external_mini_cluster.cc:1428] Setting key 8105bc86358b3a44316ef5bfbbe45fda
I20260806 16:16:10.794814 5250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:10.795754 5250 log.cc:824] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:10.797814 5250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191: No bootstrap required, opened a new log
I20260806 16:16:10.801486 5250 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a1a391c6084bc88ca42f22573d2191" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36717 } }
I20260806 16:16:10.801739 5250 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:10.801846 5250 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1a391c6084bc88ca42f22573d2191, State: Initialized, Role: FOLLOWER
I20260806 16:16:10.802418 5250 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a1a391c6084bc88ca42f22573d2191" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36717 } }
I20260806 16:16:10.802582 5250 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:10.802677 5250 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:10.802778 5250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:10.803100 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:10.795112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.18.90.62:36717)
0806 16:16:10.795792 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:10.795807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:10.795842 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:16:10.796567 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:10.796584 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:10.796920 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:10.797268 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:10.797701 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.798713 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.799620 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:10.800117 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:10.800323 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:10.801231 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:10.801236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:10.801249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:10.801623 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:10.801677 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:10.801696 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:10.801809 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:10.801969 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 16:16:10.802224 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
I20260806 16:16:10.803912 5250 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a1a391c6084bc88ca42f22573d2191" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36717 } }
I20260806 16:16:10.804316 5250 leader_election.cc:302] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3a1a391c6084bc88ca42f22573d2191; no voters:
I20260806 16:16:10.804665 5250 leader_election.cc:288] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:10.804740 5255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:10.804976 5255 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [term 1 LEADER]: Becoming Leader. State: Replica: e3a1a391c6084bc88ca42f22573d2191, State: Running, Role: LEADER
I20260806 16:16:10.805850 5255 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a1a391c6084bc88ca42f22573d2191" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36717 } }
I20260806 16:16:10.806303 5250 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:10.807485 5257 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3a1a391c6084bc88ca42f22573d2191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a1a391c6084bc88ca42f22573d2191" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36717 } } }
I20260806 16:16:10.807621 5257 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:10.807986 5256 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a1a391c6084bc88ca42f22573d2191. Latest consensus state: current_term: 1 leader_uuid: "e3a1a391c6084bc88ca42f22573d2191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a1a391c6084bc88ca42f22573d2191" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 36717 } } }
I20260806 16:16:10.808097 5264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:10.808746 5256 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:10.810896 5264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:10.815826 5264 catalog_manager.cc:1347] Generated new cluster ID: a9b37810f6d04b90bd5d222f520ab460
I20260806 16:16:10.815907 5264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:10.827598 5264 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:10.828837 5264 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:10.839097 5264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191: Generated new TSK 0
I20260806 16:16:10.839944 5264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:10.847818 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36717
--builtin_ntp_servers=127.18.90.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:16:10.961351 5274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:10.961617 5274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:10.961683 5274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:10.965829 5274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:10.965881 5274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:10.965986 5274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:16:10.970701 5274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:10.971771 5274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:10.973198 5274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:10.982587 5281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:10.982587 5279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:10.982643 5283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:10.982761 5274 server_base.cc:1034] running on GCE node
I20260806 16:16:10.983155 5274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:10.983808 5274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:10.984988 5274 hybrid_clock.cc:648] HybridClock initialized: now 1786032970984957 us; error 48 us; skew 500 ppm
I20260806 16:16:10.986720 5274 webserver.cc:460] Webserver started at http://127.18.90.1:33529/ using document root <none> and password file <none>
I20260806 16:16:10.987310 5274 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:10.987406 5274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:10.987641 5274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:10.989769 5274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "098adf1b7cac48d0a97c2815aab80f19"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "8671f531dfef1a770996ebd0662b05b8"
server_key_iv: "83259bcb893dd1754b4e7b0623af3448"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.990368 5274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "098adf1b7cac48d0a97c2815aab80f19"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "8671f531dfef1a770996ebd0662b05b8"
server_key_iv: "83259bcb893dd1754b4e7b0623af3448"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.994097 5274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:16:10.996480 5289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:10.997651 5274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:10.997797 5274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "098adf1b7cac48d0a97c2815aab80f19"
format_stamp: "Formatted at 2026-08-06 16:16:10 on dist-test-slave-0b58"
server_key: "8671f531dfef1a770996ebd0662b05b8"
server_key_iv: "83259bcb893dd1754b4e7b0623af3448"
server_key_version: "encryptionkey@0"
I20260806 16:16:10.997915 5274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:11.030153 5274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:11.030925 5274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:11.031142 5274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:11.031790 5274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:11.032828 5274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:11.032902 5274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.032977 5274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:11.033031 5274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.041968 5274 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:44899
I20260806 16:16:11.042007 5403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:44899 every 8 connection(s)
I20260806 16:16:11.042918 5274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:16:11.045143 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5274
I20260806 16:16:11.045275 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:16:11.045589 18792 external_mini_cluster.cc:1428] Setting key ac5bdf1bf5c5305d23bcc1fa4c012f92
I20260806 16:16:11.047634 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36717
--builtin_ntp_servers=127.18.90.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:16:11.053732 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.044816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:50721 (local address 127.18.90.62:36717)
0806 16:16:11.045012 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.045019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.046187 (+ 1168us) server_negotiation.cc:407] Connection header received
0806 16:16:11.047967 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.047971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.048025 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.048120 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:11.049076 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.049785 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.051556 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.051787 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.051868 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.053261 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:11.053264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:11.053266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:11.053278 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:11.053296 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:11.053304 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:11.053362 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.053558 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.053608 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 16:16:11.055002 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.045260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36717 (local address 127.18.90.1:50721)
0806 16:16:11.046019 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.046073 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.047603 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.048342 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.048349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.048433 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:11.048927 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.048939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.049937 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.049943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.051422 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.051437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.051957 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.051965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.052225 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.052932 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:11.052937 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:11.053000 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:11.053096 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:11.053395 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:11.053406 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:11.053413 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.053557 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.053878 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260806 16:16:11.056160 5404 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36717
I20260806 16:16:11.056401 5404 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:11.056929 5404 heartbeater.cc:511] Master 127.18.90.62:36717 requested a full tablet report, sending...
I20260806 16:16:11.058436 5214 ts_manager.cc:194] Registered new tserver with Master: 098adf1b7cac48d0a97c2815aab80f19 (127.18.90.1:44899)
I20260806 16:16:11.059614 5214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.1:50721
W20260806 16:16:11.159559 5407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:11.159819 5407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:11.159888 5407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:11.163954 5407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:11.164009 5407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:11.164099 5407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:16:11.168867 5407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:11.170015 5407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:11.171466 5407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:11.181140 5412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:11.181211 5416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:11.181147 5414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:11.181509 5407 server_base.cc:1034] running on GCE node
I20260806 16:16:11.181872 5407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:11.182492 5407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:11.183665 5407 hybrid_clock.cc:648] HybridClock initialized: now 1786032971183649 us; error 26 us; skew 500 ppm
I20260806 16:16:11.185572 5407 webserver.cc:460] Webserver started at http://127.18.90.2:42517/ using document root <none> and password file <none>
I20260806 16:16:11.186080 5407 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:11.186148 5407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:11.186350 5407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:11.188421 5407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "d9ae5ecc06b74ac3bd7df3df2364449a"
format_stamp: "Formatted at 2026-08-06 16:16:11 on dist-test-slave-0b58"
server_key: "dd6fedd8681c7fcfb1fd7b7669c9a6d0"
server_key_iv: "15c035f2a3a88c72138db8c1eff4fea6"
server_key_version: "encryptionkey@0"
I20260806 16:16:11.188947 5407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "d9ae5ecc06b74ac3bd7df3df2364449a"
format_stamp: "Formatted at 2026-08-06 16:16:11 on dist-test-slave-0b58"
server_key: "dd6fedd8681c7fcfb1fd7b7669c9a6d0"
server_key_iv: "15c035f2a3a88c72138db8c1eff4fea6"
server_key_version: "encryptionkey@0"
I20260806 16:16:11.192603 5407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:16:11.194803 5422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.195981 5407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:11.196121 5407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "d9ae5ecc06b74ac3bd7df3df2364449a"
format_stamp: "Formatted at 2026-08-06 16:16:11 on dist-test-slave-0b58"
server_key: "dd6fedd8681c7fcfb1fd7b7669c9a6d0"
server_key_iv: "15c035f2a3a88c72138db8c1eff4fea6"
server_key_version: "encryptionkey@0"
I20260806 16:16:11.196234 5407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:11.226735 5407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:11.227535 5407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:11.227751 5407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:11.228420 5407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:11.229377 5407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:11.229450 5407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.229523 5407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:11.229568 5407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.238237 5407 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:43131
I20260806 16:16:11.238262 5536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:43131 every 8 connection(s)
I20260806 16:16:11.239112 5407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:16:11.245416 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5407
I20260806 16:16:11.245553 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:16:11.245899 18792 external_mini_cluster.cc:1428] Setting key f745c7f2423655e59bd7515c43e38cfa
I20260806 16:16:11.248157 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36717
--builtin_ntp_servers=127.18.90.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:16:11.248966 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.240937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:33855 (local address 127.18.90.62:36717)
0806 16:16:11.241104 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.241109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.242166 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 16:16:11.243500 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.243502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.243548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.243622 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:11.244576 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.245117 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.246723 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.247066 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.247172 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.248436 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:11.248448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:11.248454 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:11.248470 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:11.248497 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:11.248510 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:11.248559 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.248793 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.248850 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":64}
I20260806 16:16:11.250142 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.241388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36717 (local address 127.18.90.2:33855)
0806 16:16:11.242024 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.242065 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.243323 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.243777 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.243783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.243864 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:11.244442 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.244456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.245247 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.245252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.246596 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.246606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.247244 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.247252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.247433 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.248074 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:11.248076 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:11.248130 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:11.248227 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:11.248627 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:11.248638 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:11.248646 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.248768 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.249071 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":119,"threads_started":1}
I20260806 16:16:11.251400 5537 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36717
I20260806 16:16:11.251626 5537 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:11.252115 5537 heartbeater.cc:511] Master 127.18.90.62:36717 requested a full tablet report, sending...
I20260806 16:16:11.253288 5214 ts_manager.cc:194] Registered new tserver with Master: d9ae5ecc06b74ac3bd7df3df2364449a (127.18.90.2:43131)
I20260806 16:16:11.254151 5214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.2:33855
W20260806 16:16:11.361634 5540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:11.361887 5540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:11.361923 5540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:11.365880 5540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:11.365972 5540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:11.366065 5540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:16:11.371151 5540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:36717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:11.372314 5540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:11.373739 5540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:11.383536 5545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:11.383545 5546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:11.383594 5540 server_base.cc:1034] running on GCE node
W20260806 16:16:11.383545 5548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:11.384194 5540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:11.384820 5540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:11.385998 5540 hybrid_clock.cc:648] HybridClock initialized: now 1786032971385964 us; error 45 us; skew 500 ppm
I20260806 16:16:11.387844 5540 webserver.cc:460] Webserver started at http://127.18.90.3:42035/ using document root <none> and password file <none>
I20260806 16:16:11.388489 5540 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:11.388579 5540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:11.388811 5540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:11.390767 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "2cb44c5a02854c5fb83cff3f79fc2668"
format_stamp: "Formatted at 2026-08-06 16:16:11 on dist-test-slave-0b58"
server_key: "0812bc9acefb958ae166f377adabd75e"
server_key_iv: "fa0564b900819a6625741b9f6dd2925d"
server_key_version: "encryptionkey@0"
I20260806 16:16:11.391315 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "2cb44c5a02854c5fb83cff3f79fc2668"
format_stamp: "Formatted at 2026-08-06 16:16:11 on dist-test-slave-0b58"
server_key: "0812bc9acefb958ae166f377adabd75e"
server_key_iv: "fa0564b900819a6625741b9f6dd2925d"
server_key_version: "encryptionkey@0"
I20260806 16:16:11.394819 5540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 16:16:11.397133 5554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.398329 5540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:11.398459 5540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "2cb44c5a02854c5fb83cff3f79fc2668"
format_stamp: "Formatted at 2026-08-06 16:16:11 on dist-test-slave-0b58"
server_key: "0812bc9acefb958ae166f377adabd75e"
server_key_iv: "fa0564b900819a6625741b9f6dd2925d"
server_key_version: "encryptionkey@0"
I20260806 16:16:11.398581 5540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:11.431223 5540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:11.431998 5540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:11.432210 5540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:11.432860 5540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:11.433807 5540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:11.433856 5540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.433897 5540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:11.433911 5540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:11.443048 5540 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:46771
I20260806 16:16:11.443066 5668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:46771 every 8 connection(s)
I20260806 16:16:11.443997 5540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:16:11.446100 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5540
I20260806 16:16:11.446244 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:16:11.446555 18792 external_mini_cluster.cc:1428] Setting key 223896b0e4d1bfa0cb4cd95d8781fd74
I20260806 16:16:11.453486 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.445850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:46629 (local address 127.18.90.62:36717)
0806 16:16:11.446002 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.446006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.447395 (+ 1389us) server_negotiation.cc:407] Connection header received
0806 16:16:11.449197 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.449200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.449246 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.449312 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:11.450125 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.450637 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.451720 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.451931 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.452004 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.453112 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:11.453114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:11.453116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:11.453126 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:11.453141 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:11.453148 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:11.453202 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.453320 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.453367 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:16:11.454393 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.446452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:36717 (local address 127.18.90.3:46629)
0806 16:16:11.447225 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.447267 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.449015 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.449442 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.449448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.449531 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:11.450012 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.450024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.450744 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.450746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.451614 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.451622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.452056 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.452059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.452197 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.452881 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:11.452882 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:11.452923 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:11.452986 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:11.453211 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:11.453216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:11.453219 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.453306 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.453560 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260806 16:16:11.455566 5669 heartbeater.cc:347] Connected to a master server at 127.18.90.62:36717
I20260806 16:16:11.455816 5669 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:11.456405 5669 heartbeater.cc:511] Master 127.18.90.62:36717 requested a full tablet report, sending...
I20260806 16:16:11.457409 5214 ts_manager.cc:194] Registered new tserver with Master: 2cb44c5a02854c5fb83cff3f79fc2668 (127.18.90.3:46771)
I20260806 16:16:11.457978 5214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.3:46629
I20260806 16:16:11.460546 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:16:11.465772 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.462449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.18.90.62:36717)
0806 16:16:11.462568 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.462575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.462727 (+ 152us) server_negotiation.cc:407] Connection header received
0806 16:16:11.462822 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.462825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.462858 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.462919 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:11.463125 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.463588 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.464429 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.464629 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.464703 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.465408 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:11.465411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:11.465413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:11.465423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:11.465440 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:11.465447 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:11.465541 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.465634 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.465665 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:11.468593 5214 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:16:11.471556 5214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:16:11.485914 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.480438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.18.90.2:43131)
0806 16:16:11.480928 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.480940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.480967 (+ 27us) server_negotiation.cc:407] Connection header received
0806 16:16:11.481043 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.481050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.481248 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.481408 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:11.481854 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.482981 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.484301 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.485119 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.485253 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.485433 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.485650 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.485749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":100,"threads_started":1}
I20260806 16:16:11.485983 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.480001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:43131 (local address 127.0.0.1:56088)
0806 16:16:11.480833 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.480843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.480934 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.481542 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.481545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.481557 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:11.481721 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.481728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.483106 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.483109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.484100 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.484110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.485259 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.485263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.485399 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.485495 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.485605 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.485815 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":118,"thread_start_us":92,"threads_started":1}
I20260806 16:16:11.485983 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.479737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:44899 (local address 127.0.0.1:44502)
0806 16:16:11.480445 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.480495 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.480629 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.481444 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.481450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.481471 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:11.481710 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.481724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.483073 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.483076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.484263 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.484289 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.485259 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.485263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.485401 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.485495 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.485606 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.485815 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":253,"spinlock_wait_cycles":10880,"thread_start_us":80,"threads_started":1}
I20260806 16:16:11.486656 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.480354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.18.90.1:44899)
0806 16:16:11.481042 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.481052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.481071 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:16:11.481132 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.481136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.481286 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.481660 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:11.481809 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.482959 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.484399 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.485127 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.485246 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.485433 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.486283 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.486494 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":150,"threads_started":1}
I20260806 16:16:11.486930 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.479655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:46771 (local address 127.0.0.1:39316)
0806 16:16:11.480446 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.480500 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.480629 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.481788 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.481792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.481812 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:11.481894 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.481898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.482880 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.482886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.484262 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.484289 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.486501 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.486504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.486599 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.486622 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.486674 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.486774 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":24,"spinlock_wait_cycles":2816,"thread_start_us":84,"threads_started":1}
I20260806 16:16:11.488987 5471 tablet_service.cc:1467] Processing CreateTablet for tablet 64ffe790f52c4b00a8247926a7763c6e (DEFAULT_TABLE table=test-table [id=8c82224193ba4e2e8fc1a701a0dbe74a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:11.489001 5338 tablet_service.cc:1467] Processing CreateTablet for tablet 64ffe790f52c4b00a8247926a7763c6e (DEFAULT_TABLE table=test-table [id=8c82224193ba4e2e8fc1a701a0dbe74a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:11.489593 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.479960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.18.90.3:46771)
0806 16:16:11.481367 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.481379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.481403 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:16:11.481483 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.481486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.481612 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.481999 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:11.482095 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.482768 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.485502 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.486356 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.487092 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.487264 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.487310 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.487397 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":96,"threads_started":1}
I20260806 16:16:11.489998 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ffe790f52c4b00a8247926a7763c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:11.490001 5471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ffe790f52c4b00a8247926a7763c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:11.492151 5602 tablet_service.cc:1467] Processing CreateTablet for tablet 64ffe790f52c4b00a8247926a7763c6e (DEFAULT_TABLE table=test-table [id=8c82224193ba4e2e8fc1a701a0dbe74a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:11.493199 5602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ffe790f52c4b00a8247926a7763c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:11.496635 5687 tablet_bootstrap.cc:492] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: Bootstrap starting.
I20260806 16:16:11.497812 5688 tablet_bootstrap.cc:492] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Bootstrap starting.
I20260806 16:16:11.498826 5687 tablet_bootstrap.cc:654] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:11.499630 5687 log.cc:824] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:11.499907 5688 tablet_bootstrap.cc:654] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:11.500325 5689 tablet_bootstrap.cc:492] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Bootstrap starting.
I20260806 16:16:11.500737 5688 log.cc:824] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:11.502326 5689 tablet_bootstrap.cc:654] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:11.502789 5688 tablet_bootstrap.cc:492] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: No bootstrap required, opened a new log
I20260806 16:16:11.502977 5688 ts_tablet_manager.cc:1397] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:16:11.502996 5689 log.cc:824] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:11.504739 5687 tablet_bootstrap.cc:492] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: No bootstrap required, opened a new log
I20260806 16:16:11.504916 5689 tablet_bootstrap.cc:492] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: No bootstrap required, opened a new log
I20260806 16:16:11.504938 5687 ts_tablet_manager.cc:1397] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260806 16:16:11.505093 5689 ts_tablet_manager.cc:1397] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:16:11.507464 5688 raft_consensus.cc:348] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.507752 5688 raft_consensus.cc:374] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:11.507815 5688 raft_consensus.cc:729] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ae5ecc06b74ac3bd7df3df2364449a, State: Initialized, Role: FOLLOWER
I20260806 16:16:11.508363 5688 consensus_queue.cc:260] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.508445 5687 raft_consensus.cc:348] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.508719 5687 raft_consensus.cc:374] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:11.508777 5687 raft_consensus.cc:729] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098adf1b7cac48d0a97c2815aab80f19, State: Initialized, Role: FOLLOWER
I20260806 16:16:11.509243 5687 consensus_queue.cc:260] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.509477 5688 ts_tablet_manager.cc:1428] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:16:11.509418 5689 raft_consensus.cc:348] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.509672 5689 raft_consensus.cc:374] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:11.509737 5689 raft_consensus.cc:729] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb44c5a02854c5fb83cff3f79fc2668, State: Initialized, Role: FOLLOWER
I20260806 16:16:11.509883 5537 heartbeater.cc:503] Master 127.18.90.62:36717 was elected leader, sending a full tablet report...
I20260806 16:16:11.510109 5689 consensus_queue.cc:260] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.510309 5687 ts_tablet_manager.cc:1428] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:16:11.510614 5404 heartbeater.cc:503] Master 127.18.90.62:36717 was elected leader, sending a full tablet report...
I20260806 16:16:11.510697 5669 heartbeater.cc:503] Master 127.18.90.62:36717 was elected leader, sending a full tablet report...
I20260806 16:16:11.510990 5689 ts_tablet_manager.cc:1428] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 16:16:11.544206 5405 tablet.cc:2369] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:16:11.589579 5693 raft_consensus.cc:482] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:11.589835 5693 raft_consensus.cc:504] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.591017 5693 leader_election.cc:288] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cb44c5a02854c5fb83cff3f79fc2668 (127.18.90.3:46771), 098adf1b7cac48d0a97c2815aab80f19 (127.18.90.1:44899)
I20260806 16:16:11.595754 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.591444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:58491 (local address 127.18.90.1:44899)
0806 16:16:11.591581 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.591587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.591847 (+ 260us) server_negotiation.cc:407] Connection header received
0806 16:16:11.591905 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.591907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.591946 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.592027 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:11.592283 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.592969 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.594488 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.595084 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.595196 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.595258 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.595548 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.595619 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:16:11.595754 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.591414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:37727 (local address 127.18.90.3:46771)
0806 16:16:11.591545 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.591549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.591566 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:16:11.591633 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.591635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.591673 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.591740 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:11.592073 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.592888 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.594488 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.595084 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.595208 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.595258 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.595548 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.595619 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:16:11.595773 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.591229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:46771 (local address 127.18.90.2:37727)
0806 16:16:11.591345 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.591364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.591459 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.591786 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.591791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.591812 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:11.591946 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.591951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.592989 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.592991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.594353 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.594362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.595206 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.595218 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.595381 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.595441 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.595516 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.595609 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":128}
I20260806 16:16:11.595782 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.591294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:44899 (local address 127.18.90.2:58491)
0806 16:16:11.591735 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.591749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:11.591833 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:11.592039 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:11.592043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:11.592061 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:11.592154 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.592157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.593057 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.593060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.594351 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:11.594362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.595206 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.595209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:11.595377 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.595441 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 16:16:11.595516 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:16:11.595609 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":132,"thread_start_us":140,"threads_started":1}
I20260806 16:16:11.596542 5622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ffe790f52c4b00a8247926a7763c6e" candidate_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" is_pre_election: true
I20260806 16:16:11.596683 5358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ffe790f52c4b00a8247926a7763c6e" candidate_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098adf1b7cac48d0a97c2815aab80f19" is_pre_election: true
I20260806 16:16:11.596910 5622 raft_consensus.cc:2454] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ae5ecc06b74ac3bd7df3df2364449a in term 0.
I20260806 16:16:11.597071 5358 raft_consensus.cc:2454] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ae5ecc06b74ac3bd7df3df2364449a in term 0.
I20260806 16:16:11.597616 5425 leader_election.cc:302] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 098adf1b7cac48d0a97c2815aab80f19, 2cb44c5a02854c5fb83cff3f79fc2668, d9ae5ecc06b74ac3bd7df3df2364449a; no voters:
I20260806 16:16:11.597919 5693 raft_consensus.cc:2781] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:16:11.598023 5693 raft_consensus.cc:482] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:16:11.598105 5693 raft_consensus.cc:3037] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:11.599162 5693 raft_consensus.cc:504] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.599558 5693 leader_election.cc:288] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [CANDIDATE]: Term 1 election: Requested vote from peers 2cb44c5a02854c5fb83cff3f79fc2668 (127.18.90.3:46771), 098adf1b7cac48d0a97c2815aab80f19 (127.18.90.1:44899)
I20260806 16:16:11.599951 5358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ffe790f52c4b00a8247926a7763c6e" candidate_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098adf1b7cac48d0a97c2815aab80f19"
I20260806 16:16:11.600086 5358 raft_consensus.cc:3037] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:11.601402 5358 raft_consensus.cc:2454] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ae5ecc06b74ac3bd7df3df2364449a in term 1.
I20260806 16:16:11.601789 5425 leader_election.cc:302] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098adf1b7cac48d0a97c2815aab80f19, d9ae5ecc06b74ac3bd7df3df2364449a; no voters:
I20260806 16:16:11.601769 5622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ffe790f52c4b00a8247926a7763c6e" candidate_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb44c5a02854c5fb83cff3f79fc2668"
I20260806 16:16:11.601899 5622 raft_consensus.cc:3037] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:11.601964 5693 raft_consensus.cc:2781] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:11.602210 5693 raft_consensus.cc:686] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 1 LEADER]: Becoming Leader. State: Replica: d9ae5ecc06b74ac3bd7df3df2364449a, State: Running, Role: LEADER
I20260806 16:16:11.602587 5693 consensus_queue.cc:237] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } }
I20260806 16:16:11.603227 5622 raft_consensus.cc:2454] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ae5ecc06b74ac3bd7df3df2364449a in term 1.
I20260806 16:16:11.606122 5213 catalog_manager.cc:5495] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a reported cstate change: term changed from 0 to 1, leader changed from <none> to d9ae5ecc06b74ac3bd7df3df2364449a (127.18.90.2). New cstate: current_term: 1 leader_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ae5ecc06b74ac3bd7df3df2364449a" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 43131 } health_report { overall_health: HEALTHY } } }
I20260806 16:16:11.639309 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.635630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.18.90.2:43131)
0806 16:16:11.635781 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:11.635785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:11.635822 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:16:11.635906 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:11.635909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:11.635965 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:11.636083 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:11.636372 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.636882 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.637756 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:11.637971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:11.638081 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:11.638425 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:11.638842 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:11.639010 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:11.639096 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 16:16:11.639134 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:16:11.645538 5622 raft_consensus.cc:1261] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 1 FOLLOWER]: Refusing update from remote peer d9ae5ecc06b74ac3bd7df3df2364449a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:11.645538 5358 raft_consensus.cc:1261] T 64ffe790f52c4b00a8247926a7763c6e P 098adf1b7cac48d0a97c2815aab80f19 [term 1 FOLLOWER]: Refusing update from remote peer d9ae5ecc06b74ac3bd7df3df2364449a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:11.646382 5693 consensus_queue.cc:1035] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb44c5a02854c5fb83cff3f79fc2668" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:11.646659 5697 consensus_queue.cc:1035] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "098adf1b7cac48d0a97c2815aab80f19" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:11.654256 5700 mvcc.cc:205] Tried to move back new op lower bound from 7315591051852210176 to 7315591051687727104. Current Snapshot: MvccSnapshot[applied={T|T < 7315591051852210176}]
I20260806 16:16:11.655794 5701 mvcc.cc:205] Tried to move back new op lower bound from 7315591051852210176 to 7315591051687727104. Current Snapshot: MvccSnapshot[applied={T|T < 7315591051852210176}]
I20260806 16:16:11.662608 5214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:11.662765 5214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:11.665129 5214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3a1a391c6084bc88ca42f22573d2191: Sending DeleteTablet for 3 replicas of tablet 64ffe790f52c4b00a8247926a7763c6e
I20260806 16:16:11.665923 5602 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ffe790f52c4b00a8247926a7763c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:11 UTC) from {username='slave'} at 127.0.0.1:39316
I20260806 16:16:11.666117 5338 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ffe790f52c4b00a8247926a7763c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:11 UTC) from {username='slave'} at 127.0.0.1:44502
I20260806 16:16:11.666589 5471 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ffe790f52c4b00a8247926a7763c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:11 UTC) from {username='slave'} at 127.0.0.1:56088
I20260806 16:16:11.666923 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5274
I20260806 16:16:11.672544 5712 tablet_replica.cc:330] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: stopping tablet replica
I20260806 16:16:11.672551 5710 tablet_replica.cc:330] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: stopping tablet replica
I20260806 16:16:11.672894 5712 raft_consensus.cc:2229] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:16:11.672919 5710 raft_consensus.cc:2229] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:11.673259 5712 raft_consensus.cc:2258] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:11.673260 5710 raft_consensus.cc:2258] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:11.674846 5712 ts_tablet_manager.cc:1905] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:11.675033 5710 ts_tablet_manager.cc:1905] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 16:16:11.675881 5200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:44899 (error 108)
I20260806 16:16:11.677475 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.677109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:44899 (local address 127.0.0.1:44512)
0806 16:16:11.677317 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.677389 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260806 16:16:11.677659 5200 catalog_manager.cc:4584] TS 098adf1b7cac48d0a97c2815aab80f19 (127.18.90.1:44899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64ffe790f52c4b00a8247926a7763c6e: Network error: Client connection negotiation failed: client connection to 127.18.90.1:44899: connect: Connection refused (error 111)
I20260806 16:16:11.677911 5712 ts_tablet_manager.cc:1918] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:11.677973 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5407
I20260806 16:16:11.678012 5712 log.cc:1196] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/64ffe790f52c4b00a8247926a7763c6e
I20260806 16:16:11.678336 5712 ts_tablet_manager.cc:1939] T 64ffe790f52c4b00a8247926a7763c6e P d9ae5ecc06b74ac3bd7df3df2364449a: Deleting consensus metadata
I20260806 16:16:11.679160 5710 ts_tablet_manager.cc:1918] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:11.679314 5710 log.cc:1196] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/64ffe790f52c4b00a8247926a7763c6e
I20260806 16:16:11.679677 5710 ts_tablet_manager.cc:1939] T 64ffe790f52c4b00a8247926a7763c6e P 2cb44c5a02854c5fb83cff3f79fc2668: Deleting consensus metadata
I20260806 16:16:11.681023 5199 catalog_manager.cc:4841] TS 2cb44c5a02854c5fb83cff3f79fc2668 (127.18.90.3:46771): tablet 64ffe790f52c4b00a8247926a7763c6e (table test-table [id=8c82224193ba4e2e8fc1a701a0dbe74a]) successfully deleted
I20260806 16:16:11.685022 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5540
I20260806 16:16:11.685529 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:11.685163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:43131 (local address 127.0.0.1:56106)
0806 16:16:11.685368 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:11.685428 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.2:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 16:16:11.691195 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5183
2026-08-06T16:16:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1115 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:11 dist-test-slave-0b58 krb5kdc[5716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:11 dist-test-slave-0b58 krb5kdc[5716](info): set up 2 sockets
Aug 06 16:16:11 dist-test-slave-0b58 krb5kdc[5716](info): commencing operation
krb5kdc: starting...
W20260806 16:16:12.732547 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:12 dist-test-slave-0b58 krb5kdc[5716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:12.857255 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44421
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:44421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:12.969612 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:12.969928 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:12.970014 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:12.974037 5729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:12.974120 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:12.974160 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:12.974225 5729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:12.974265 5729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:12.979449 5729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:44421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:12.980587 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:12.981613 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:12.989912 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:12.989961 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:12.989934 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:12.990336 5729 server_base.cc:1034] running on GCE node
I20260806 16:16:12.990878 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:12.991961 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:12.993170 5729 hybrid_clock.cc:648] HybridClock initialized: now 1786032972993135 us; error 47 us; skew 500 ppm
Aug 06 16:16:12 dist-test-slave-0b58 krb5kdc[5716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032972, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:12.996027 5729 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:12.996934 5729 webserver.cc:460] Webserver started at http://127.18.90.62:36675/ using document root <none> and password file <none>
I20260806 16:16:12.997454 5729 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:12.997512 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:12.997671 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:12.999620 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "baa7844fcef144769ef059ff0f6318ba"
format_stamp: "Formatted at 2026-08-06 16:16:12 on dist-test-slave-0b58"
server_key: "f70f0eca281853fe7eb4471f2bf893e7"
server_key_iv: "e12dbb4437473ab3b18c0131b5f9423c"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.000144 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "baa7844fcef144769ef059ff0f6318ba"
format_stamp: "Formatted at 2026-08-06 16:16:12 on dist-test-slave-0b58"
server_key: "f70f0eca281853fe7eb4471f2bf893e7"
server_key_iv: "e12dbb4437473ab3b18c0131b5f9423c"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.003850 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:16:13.006645 5744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.007905 5729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:13.008015 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "baa7844fcef144769ef059ff0f6318ba"
format_stamp: "Formatted at 2026-08-06 16:16:12 on dist-test-slave-0b58"
server_key: "f70f0eca281853fe7eb4471f2bf893e7"
server_key_iv: "e12dbb4437473ab3b18c0131b5f9423c"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.008114 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:13.024367 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:13.027679 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:13.027839 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:13.035004 5729 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:44421
I20260806 16:16:13.035025 5796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:44421 every 8 connection(s)
I20260806 16:16:13.035964 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:13.038805 5798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:13.043462 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5729
I20260806 16:16:13.043623 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:13.043957 18792 external_mini_cluster.cc:1428] Setting key dd2524e0023279d4549e6d3501d2b9cd
I20260806 16:16:13.044792 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba: Bootstrap starting.
I20260806 16:16:13.047194 5798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:13.047942 5798 log.cc:824] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:13.049898 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba: No bootstrap required, opened a new log
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:13.053604 5798 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa7844fcef144769ef059ff0f6318ba" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44421 } }
I20260806 16:16:13.053896 5798 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:13.053972 5798 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa7844fcef144769ef059ff0f6318ba, State: Initialized, Role: FOLLOWER
I20260806 16:16:13.054486 5798 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa7844fcef144769ef059ff0f6318ba" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44421 } }
I20260806 16:16:13.054682 5798 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:13.054759 5798 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:13.054858 5798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:13.055714 5798 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa7844fcef144769ef059ff0f6318ba" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44421 } }
I20260806 16:16:13.056067 5798 leader_election.cc:302] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa7844fcef144769ef059ff0f6318ba; no voters:
I20260806 16:16:13.056423 5798 leader_election.cc:288] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:13.056519 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:13.056773 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [term 1 LEADER]: Becoming Leader. State: Replica: baa7844fcef144769ef059ff0f6318ba, State: Running, Role: LEADER
I20260806 16:16:13.057102 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa7844fcef144769ef059ff0f6318ba" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44421 } }
I20260806 16:16:13.057844 5798 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:13.058748 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.045481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.18.90.62:44421)
0806 16:16:13.045902 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.045917 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.045953 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:16:13.046584 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.046602 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.046937 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.047281 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:13.048112 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.049358 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.050297 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.050767 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.050943 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.053442 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:13.053447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:13.053459 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:13.053485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:13.055371 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.055810 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.055816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.055822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.056004 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.056338 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.056342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.056344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.056900 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:13.057143 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.057457 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.057755 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260806 16:16:13.058784 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa7844fcef144769ef059ff0f6318ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa7844fcef144769ef059ff0f6318ba" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44421 } } }
I20260806 16:16:13.058921 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:13.059271 5812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:13.060171 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa7844fcef144769ef059ff0f6318ba. Latest consensus state: current_term: 1 leader_uuid: "baa7844fcef144769ef059ff0f6318ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa7844fcef144769ef059ff0f6318ba" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44421 } } }
I20260806 16:16:13.060360 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:13.062215 5812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:13.067909 5812 catalog_manager.cc:1347] Generated new cluster ID: aede5058fbc84d0ea5cd1f1d6dae3c9a
I20260806 16:16:13.067979 5812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:13.077589 5812 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:13.078547 5812 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:13.088788 5812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba: Generated new TSK 0
I20260806 16:16:13.089663 5812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.90.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
Entry for principal kudu/127.18.90.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab.
I20260806 16:16:13.119091 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44421
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:13.233781 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:13.234050 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:13.234112 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:13.238233 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:13.238297 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:13.238358 5825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:16:13.243522 5825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.1.keytab
--principal=kudu/127.18.90.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:13.244709 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:13.245765 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:13.255506 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:13.255508 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:13.255508 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:13.255728 5825 server_base.cc:1034] running on GCE node
I20260806 16:16:13.256206 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:13.256873 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:13.258036 5825 hybrid_clock.cc:648] HybridClock initialized: now 1786032973258019 us; error 30 us; skew 500 ppm
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:13.260960 5825 init.cc:377] Logged in from keytab as kudu/127.18.90.1@KRBTEST.COM (short username kudu)
I20260806 16:16:13.261765 5825 webserver.cc:460] Webserver started at http://127.18.90.1:35215/ using document root <none> and password file <none>
I20260806 16:16:13.262275 5825 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:13.262332 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:13.262504 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:13.264528 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "5c8d6f8f274847749163286bca28e908"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "8808fad13e152e97e0ff96ab0de82271"
server_key_iv: "affef46a50c1488bd340f4952490c04b"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.265045 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "5c8d6f8f274847749163286bca28e908"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "8808fad13e152e97e0ff96ab0de82271"
server_key_iv: "affef46a50c1488bd340f4952490c04b"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.268833 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:16:13.271260 5840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.272588 5825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:13.272729 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "5c8d6f8f274847749163286bca28e908"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "8808fad13e152e97e0ff96ab0de82271"
server_key_iv: "affef46a50c1488bd340f4952490c04b"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.272855 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:13.289132 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:13.292178 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:13.292419 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:13.293026 5825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:13.293905 5825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:13.293952 5825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.293994 5825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:13.294025 5825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.303292 5825 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:35475
I20260806 16:16:13.303313 5956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:35475 every 8 connection(s)
I20260806 16:16:13.304206 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:16:13.306737 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5825
I20260806 16:16:13.306882 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:16:13.307260 18792 external_mini_cluster.cc:1428] Setting key a222d0fb143f04bdcad5bc8127c2085b
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.1@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.2@KRBTEST.COM; defaulting to no policy
I20260806 16:16:13.318454 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.306049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:36413 (local address 127.18.90.62:44421)
0806 16:16:13.306200 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.306204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.307198 (+ 994us) server_negotiation.cc:407] Connection header received
0806 16:16:13.308785 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.308789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.308842 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.308921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:13.310077 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.310615 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.311890 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.312128 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.312237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.314806 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:13.314809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:13.314811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:13.314835 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:13.316316 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.316824 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.316829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.316831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.316916 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.317278 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.317281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.317283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.317495 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:13.317606 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.318110 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.318243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:13.319368 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.306535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44421 (local address 127.18.90.1:36413)
0806 16:16:13.307015 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:13.307075 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:13.308497 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:13.309091 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:13.309098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:13.309460 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:13.309934 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.309947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.310750 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.310755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.311727 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.311762 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.312324 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.312331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.312517 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.313160 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:13.313177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:13.314630 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:13.316434 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:13.316440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:13.316456 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:13.316718 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:13.317012 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:13.317015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:13.317017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:13.317165 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:13.317606 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:13.317614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:13.317867 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 16:16:13.318057 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 16:16:13.318475 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.2@KRBTEST.COM" created.
I20260806 16:16:13.320710 5957 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44421
I20260806 16:16:13.320986 5957 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:13.321511 5957 heartbeater.cc:511] Master 127.18.90.62:44421 requested a full tablet report, sending...
I20260806 16:16:13.323269 5761 ts_manager.cc:194] Registered new tserver with Master: 5c8d6f8f274847749163286bca28e908 (127.18.90.1:35475)
I20260806 16:16:13.324571 5761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.1@KRBTEST.COM'} at 127.18.90.1:36413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
Entry for principal kudu/127.18.90.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab.
I20260806 16:16:13.335700 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44421
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:13.447199 5963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:13.447510 5963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:13.447587 5963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:13.451808 5963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:13.451882 5963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:13.451972 5963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:16:13.457118 5963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.2.keytab
--principal=kudu/127.18.90.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:13.458240 5963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:13.459338 5963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:13.469233 5969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:13.469233 5968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:13.469233 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:13.469579 5963 server_base.cc:1034] running on GCE node
I20260806 16:16:13.469981 5963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:13.470662 5963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:13.471868 5963 hybrid_clock.cc:648] HybridClock initialized: now 1786032973471826 us; error 51 us; skew 500 ppm
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:13.474738 5963 init.cc:377] Logged in from keytab as kudu/127.18.90.2@KRBTEST.COM (short username kudu)
I20260806 16:16:13.475723 5963 webserver.cc:460] Webserver started at http://127.18.90.2:44163/ using document root <none> and password file <none>
I20260806 16:16:13.476388 5963 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:13.476483 5963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:13.476694 5963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:13.478819 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "c6b5703f19c54f4d98d37b7ea88d9e09"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "497fb60bb85f6844881d8ecaef367be1"
server_key_iv: "a610ccb0699c015b656b444ddb7168bd"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.479419 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b5703f19c54f4d98d37b7ea88d9e09"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "497fb60bb85f6844881d8ecaef367be1"
server_key_iv: "a610ccb0699c015b656b444ddb7168bd"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.483186 5963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 16:16:13.485637 5978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.486792 5963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:13.486923 5963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "c6b5703f19c54f4d98d37b7ea88d9e09"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "497fb60bb85f6844881d8ecaef367be1"
server_key_iv: "a610ccb0699c015b656b444ddb7168bd"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.487032 5963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:13.518280 5963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:13.521864 5963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:13.522079 5963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:13.522748 5963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:13.523661 5963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:13.523717 5963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.523819 5963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:13.523856 5963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.533178 5963 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:44025
I20260806 16:16:13.533197 6093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:44025 every 8 connection(s)
I20260806 16:16:13.534125 5963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:16:13.543304 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 5963
I20260806 16:16:13.543445 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:16:13.543730 18792 external_mini_cluster.cc:1428] Setting key 63559c219275426ea237a4e0c51c51cb
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.2@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:13.548509 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.535932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:54115 (local address 127.18.90.62:44421)
0806 16:16:13.536073 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.536076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.536967 (+ 891us) server_negotiation.cc:407] Connection header received
0806 16:16:13.538350 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.538354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.538416 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.538518 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:13.539702 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.540235 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.541442 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.541696 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.541818 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.544574 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:13.544581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:13.544583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:13.544614 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:13.546073 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.546725 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.546727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.546728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.546806 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.547197 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.547199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.547200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.547376 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:13.547450 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.548120 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.548287 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:16:13.549304 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.536339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44421 (local address 127.18.90.2:54115)
0806 16:16:13.536805 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:13.536851 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:13.538111 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:13.538698 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:13.538705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:13.539064 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:13.539546 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.539559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.540416 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.540420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.541310 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.541320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.541872 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.541876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.542045 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.542694 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:13.542710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:13.544363 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:13.546202 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:13.546212 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:13.546230 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:13.546615 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:13.546896 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:13.546902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:13.546907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:13.547101 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:13.547480 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:13.547489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:13.547852 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 16:16:13.548034 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 16:16:13.548356 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260806 16:16:13.550638 6094 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44421
I20260806 16:16:13.550902 6094 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:13.551388 6094 heartbeater.cc:511] Master 127.18.90.62:44421 requested a full tablet report, sending...
I20260806 16:16:13.552541 5761 ts_manager.cc:194] Registered new tserver with Master: c6b5703f19c54f4d98d37b7ea88d9e09 (127.18.90.2:44025)
I20260806 16:16:13.553387 5761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.2@KRBTEST.COM'} at 127.18.90.2:54115
WARNING: no policy specified for kudu/127.18.90.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
Entry for principal kudu/127.18.90.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab.
I20260806 16:16:13.572880 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44421
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:13.686491 6100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:13.686800 6100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:13.686877 6100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:13.691155 6100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:13.691252 6100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:13.691349 6100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:16:13.696798 6100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.3.keytab
--principal=kudu/127.18.90.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:13.697923 6100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:13.699041 6100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:13.708758 6106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:13.708781 6108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:13.709017 6100 server_base.cc:1034] running on GCE node
W20260806 16:16:13.708758 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:13.709494 6100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:13.710165 6100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:13.711328 6100 hybrid_clock.cc:648] HybridClock initialized: now 1786032973711297 us; error 47 us; skew 500 ppm
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:13.714190 6100 init.cc:377] Logged in from keytab as kudu/127.18.90.3@KRBTEST.COM (short username kudu)
I20260806 16:16:13.715167 6100 webserver.cc:460] Webserver started at http://127.18.90.3:44441/ using document root <none> and password file <none>
I20260806 16:16:13.715742 6100 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:13.715838 6100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:13.716074 6100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:13.718207 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "f2d02fa593a2491895fc346d4cfad2f8"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "30e19923ca1202f26a24a56360d117b0"
server_key_iv: "11b5ab94da6b80d6dc25101dea5362b4"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.718801 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "f2d02fa593a2491895fc346d4cfad2f8"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "30e19923ca1202f26a24a56360d117b0"
server_key_iv: "11b5ab94da6b80d6dc25101dea5362b4"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.722703 6100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 16:16:13.725203 6115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.726567 6100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:13.726709 6100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "f2d02fa593a2491895fc346d4cfad2f8"
format_stamp: "Formatted at 2026-08-06 16:16:13 on dist-test-slave-0b58"
server_key: "30e19923ca1202f26a24a56360d117b0"
server_key_iv: "11b5ab94da6b80d6dc25101dea5362b4"
server_key_version: "encryptionkey@0"
I20260806 16:16:13.726830 6100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:13.748754 6100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:13.751698 6100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:13.751940 6100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:13.752676 6100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:13.753531 6100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:13.753603 6100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.753676 6100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:13.753731 6100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:13.763178 6100 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:44109
I20260806 16:16:13.763214 6231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:44109 every 8 connection(s)
I20260806 16:16:13.764092 6100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:16:13.770162 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6100
I20260806 16:16:13.770334 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:16:13.770690 18792 external_mini_cluster.cc:1428] Setting key 1acbb309e03828d8400e8f494afb3d9a
Aug 06 16:16:13 dist-test-slave-0b58 krb5kdc[5716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.3@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:13.778318 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.765971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:38221 (local address 127.18.90.62:44421)
0806 16:16:13.766112 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.766115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.766940 (+ 825us) server_negotiation.cc:407] Connection header received
0806 16:16:13.768228 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.768231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.768333 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.768430 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:13.769642 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.770173 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.771810 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.772128 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.772237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.774804 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:13.774806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:13.774809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:13.774834 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:13.776347 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.776921 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.776924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.776925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.777003 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.777310 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.777312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.777314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.777483 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:13.777558 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.778078 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.778174 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:16:13.779146 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.766322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:44421 (local address 127.18.90.3:38221)
0806 16:16:13.766773 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:13.766818 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:13.768022 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:13.768645 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:13.768652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:13.769008 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:13.769506 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.769520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.770326 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.770330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.771681 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.771702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.772336 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.772340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.772493 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.773150 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:13.773167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:13.774641 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:13.776444 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:13.776449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:13.776461 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:13.776834 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:13.777087 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:13.777089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:13.777091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:13.777229 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:13.777571 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:13.777576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:13.777863 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 16:16:13.778024 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 16:16:13.778296 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260806 16:16:13.780359 6232 heartbeater.cc:347] Connected to a master server at 127.18.90.62:44421
I20260806 16:16:13.780607 6232 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:13.781095 6232 heartbeater.cc:511] Master 127.18.90.62:44421 requested a full tablet report, sending...
I20260806 16:16:13.782191 5761 ts_manager.cc:194] Registered new tserver with Master: f2d02fa593a2491895fc346d4cfad2f8 (127.18.90.3:44109)
I20260806 16:16:13.782752 5761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.90.3@KRBTEST.COM'} at 127.18.90.3:38221
I20260806 16:16:13.784693 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:16:13.793725 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.786738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.18.90.62:44421)
0806 16:16:13.786859 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.786863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.787053 (+ 190us) server_negotiation.cc:407] Connection header received
0806 16:16:13.787138 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.787141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.787175 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.787291 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:13.787934 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.788501 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.789288 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.789487 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.789560 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.791064 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:13.791067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:13.791069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:13.791088 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:13.792326 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.792698 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.792700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.792701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.792775 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:13.793000 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:13.793002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:13.793003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:13.793198 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:13.793299 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.793514 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.793603 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:13.796624 5761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:16:13.799592 5761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:16:13.813995 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.807921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:44109 (local address 127.0.0.1:35906)
0806 16:16:13.808683 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:13.808695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:13.808761 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:13.809768 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:13.809771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:13.809789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:13.809951 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.809958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.810988 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.810993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.812419 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.812431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.813421 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.813424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.813532 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.813568 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 16:16:13.813679 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 16:16:13.813817 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":107,"thread_start_us":91,"threads_started":1}
I20260806 16:16:13.814170 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.808148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44025 (local address 127.0.0.1:43984)
0806 16:16:13.808975 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:13.808987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:13.809057 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:13.809596 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:13.809598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:13.809624 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:13.810002 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.810029 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.811388 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.811392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.812565 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.812575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.813672 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.813676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.813793 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.813814 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:16:13.813896 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 16:16:13.813953 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":299,"spinlock_wait_cycles":5632,"thread_start_us":284,"threads_started":1}
I20260806 16:16:13.814388 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.807742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35475 (local address 127.0.0.1:43526)
0806 16:16:13.808579 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:13.808632 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:13.808769 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:13.809578 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:13.809586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:13.809619 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:13.809778 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.809786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.811284 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.811288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.812706 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:13.812715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.813686 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.813689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:13.814078 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.814100 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:13.814174 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 16:16:13.814255 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":377,"spinlock_wait_cycles":10240,"thread_start_us":86,"threads_started":1}
I20260806 16:16:13.814445 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.808593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.18.90.2:44025)
0806 16:16:13.809161 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.809173 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.809196 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:16:13.809276 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.809286 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.809431 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.809778 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:13.810149 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.811240 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.812689 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.813505 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.813670 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.813830 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.813930 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.814030 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":116,"threads_started":1}
I20260806 16:16:13.814697 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.807914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.18.90.1:35475)
0806 16:16:13.809040 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.809055 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.809080 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:16:13.809165 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.809171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.809397 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.809556 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:13.809895 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.811123 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.812805 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.813552 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.813857 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.814004 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.814431 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.814548 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":89,"threads_started":1}
I20260806 16:16:13.815344 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:13.808971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.18.90.3:44109)
0806 16:16:13.809294 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:13.809305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:13.809326 (+ 21us) server_negotiation.cc:407] Connection header received
0806 16:16:13.809395 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:13.809400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:13.809560 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:13.809695 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:13.810078 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.810851 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.812579 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:13.813294 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:13.813420 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:13.813577 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:13.814213 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 16:16:13.815170 (+ 957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260806 16:16:13.817148 5889 tablet_service.cc:1467] Processing CreateTablet for tablet 2553b614cac74870ae319e2280b195e8 (DEFAULT_TABLE table=test-table [id=cf7fa40dba224e2fbec3f3a8e9f18d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:13.817144 6026 tablet_service.cc:1467] Processing CreateTablet for tablet 2553b614cac74870ae319e2280b195e8 (DEFAULT_TABLE table=test-table [id=cf7fa40dba224e2fbec3f3a8e9f18d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:13.818126 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2553b614cac74870ae319e2280b195e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:13.818197 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2553b614cac74870ae319e2280b195e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:13.820226 6164 tablet_service.cc:1467] Processing CreateTablet for tablet 2553b614cac74870ae319e2280b195e8 (DEFAULT_TABLE table=test-table [id=cf7fa40dba224e2fbec3f3a8e9f18d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:13.821237 6164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2553b614cac74870ae319e2280b195e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:13.824678 6249 tablet_bootstrap.cc:492] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: Bootstrap starting.
I20260806 16:16:13.826969 6249 tablet_bootstrap.cc:654] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:13.827811 6249 log.cc:824] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:13.828008 6250 tablet_bootstrap.cc:492] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Bootstrap starting.
I20260806 16:16:13.829133 6251 tablet_bootstrap.cc:492] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Bootstrap starting.
I20260806 16:16:13.830499 6250 tablet_bootstrap.cc:654] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:13.830641 6249 tablet_bootstrap.cc:492] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: No bootstrap required, opened a new log
I20260806 16:16:13.830870 6249 ts_tablet_manager.cc:1397] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:13.831332 6251 tablet_bootstrap.cc:654] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:13.831916 6250 log.cc:824] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:13.832624 6251 log.cc:824] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:13.833973 6250 tablet_bootstrap.cc:492] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: No bootstrap required, opened a new log
I20260806 16:16:13.834172 6250 ts_tablet_manager.cc:1397] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260806 16:16:13.834648 6251 tablet_bootstrap.cc:492] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: No bootstrap required, opened a new log
I20260806 16:16:13.834821 6251 ts_tablet_manager.cc:1397] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:13.835017 6249 raft_consensus.cc:348] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:13.835279 6249 raft_consensus.cc:374] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:13.835353 6249 raft_consensus.cc:729] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8d6f8f274847749163286bca28e908, State: Initialized, Role: FOLLOWER
I20260806 16:16:13.835904 6249 consensus_queue.cc:260] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:13.837046 6249 ts_tablet_manager.cc:1428] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 16:16:13.837440 5957 heartbeater.cc:503] Master 127.18.90.62:44421 was elected leader, sending a full tablet report...
I20260806 16:16:13.838093 6250 raft_consensus.cc:348] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:13.838377 6250 raft_consensus.cc:374] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:13.838479 6250 raft_consensus.cc:729] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b5703f19c54f4d98d37b7ea88d9e09, State: Initialized, Role: FOLLOWER
I20260806 16:16:13.838596 6251 raft_consensus.cc:348] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:13.838855 6251 raft_consensus.cc:374] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:13.838917 6251 raft_consensus.cc:729] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d02fa593a2491895fc346d4cfad2f8, State: Initialized, Role: FOLLOWER
I20260806 16:16:13.838953 6250 consensus_queue.cc:260] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:13.839337 6251 consensus_queue.cc:260] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:13.839612 6094 heartbeater.cc:503] Master 127.18.90.62:44421 was elected leader, sending a full tablet report...
I20260806 16:16:13.839948 6250 ts_tablet_manager.cc:1428] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260806 16:16:13.840447 6251 ts_tablet_manager.cc:1428] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:16:13.840818 6232 heartbeater.cc:503] Master 127.18.90.62:44421 was elected leader, sending a full tablet report...
W20260806 16:16:14.015132 6233 tablet.cc:2369] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:16:14.035346 6095 tablet.cc:2369] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:16:14.055605 5958 tablet.cc:2369] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:16:14.062492 6257 raft_consensus.cc:482] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:14.062729 6257 raft_consensus.cc:504] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:14.063833 6257 leader_election.cc:288] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c8d6f8f274847749163286bca28e908 (127.18.90.1:35475), c6b5703f19c54f4d98d37b7ea88d9e09 (127.18.90.2:44025)
I20260806 16:16:14.067843 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.064112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44025 (local address 127.18.90.3:56833)
0806 16:16:14.064244 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:14.064283 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:14.064386 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:14.064686 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:14.064688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:14.064706 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:14.064875 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.064882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.065756 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.065759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.066631 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.066640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.067406 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.067410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.067520 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.067547 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:16:14.067627 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:16:14.067711 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260806 16:16:14.067883 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.064213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:56833 (local address 127.18.90.2:44025)
0806 16:16:14.064399 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:14.064405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:14.064423 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:16:14.064513 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:14.064518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:14.064572 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:14.064671 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:14.064994 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.065649 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.066767 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.067285 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.067369 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.067424 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:14.067667 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 16:16:14.067743 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 16:16:14.068110 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.064118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35475 (local address 127.18.90.3:39211)
0806 16:16:14.064666 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:14.064679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:14.064778 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:14.065057 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:14.065060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:14.065078 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:14.065216 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.065223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.065994 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.065997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.066909 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.066919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.067682 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.067685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.067781 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.067803 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:14.067882 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:16:14.067954 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":150,"thread_start_us":142,"threads_started":1}
I20260806 16:16:14.068146 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.064241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:39211 (local address 127.18.90.1:35475)
0806 16:16:14.064384 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:14.064389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:14.064765 (+ 376us) server_negotiation.cc:407] Connection header received
0806 16:16:14.064891 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:14.064894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:14.064953 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:14.065029 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:14.065334 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.065856 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.067032 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.067543 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.067650 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.067708 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:14.067918 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 16:16:14.067999 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 16:16:14.068800 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2553b614cac74870ae319e2280b195e8" candidate_uuid: "f2d02fa593a2491895fc346d4cfad2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" is_pre_election: true
I20260806 16:16:14.068917 5910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2553b614cac74870ae319e2280b195e8" candidate_uuid: "f2d02fa593a2491895fc346d4cfad2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8d6f8f274847749163286bca28e908" is_pre_election: true
I20260806 16:16:14.069159 6047 raft_consensus.cc:2454] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d02fa593a2491895fc346d4cfad2f8 in term 0.
I20260806 16:16:14.069260 5910 raft_consensus.cc:2454] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d02fa593a2491895fc346d4cfad2f8 in term 0.
I20260806 16:16:14.069681 6116 leader_election.cc:302] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b5703f19c54f4d98d37b7ea88d9e09, f2d02fa593a2491895fc346d4cfad2f8; no voters:
I20260806 16:16:14.069993 6257 raft_consensus.cc:2781] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:16:14.070084 6257 raft_consensus.cc:482] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:16:14.070144 6257 raft_consensus.cc:3037] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:14.071197 6257 raft_consensus.cc:504] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:14.071640 6257 leader_election.cc:288] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [CANDIDATE]: Term 1 election: Requested vote from peers 5c8d6f8f274847749163286bca28e908 (127.18.90.1:35475), c6b5703f19c54f4d98d37b7ea88d9e09 (127.18.90.2:44025)
I20260806 16:16:14.072009 5910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2553b614cac74870ae319e2280b195e8" candidate_uuid: "f2d02fa593a2491895fc346d4cfad2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8d6f8f274847749163286bca28e908"
I20260806 16:16:14.072081 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2553b614cac74870ae319e2280b195e8" candidate_uuid: "f2d02fa593a2491895fc346d4cfad2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09"
I20260806 16:16:14.072160 5910 raft_consensus.cc:3037] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:14.072214 6047 raft_consensus.cc:3037] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:14.073242 5910 raft_consensus.cc:2454] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d02fa593a2491895fc346d4cfad2f8 in term 1.
I20260806 16:16:14.073333 6047 raft_consensus.cc:2454] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d02fa593a2491895fc346d4cfad2f8 in term 1.
I20260806 16:16:14.073554 6119 leader_election.cc:302] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8d6f8f274847749163286bca28e908, f2d02fa593a2491895fc346d4cfad2f8; no voters:
I20260806 16:16:14.073760 6257 raft_consensus.cc:2781] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:14.074016 6257 raft_consensus.cc:686] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 1 LEADER]: Becoming Leader. State: Replica: f2d02fa593a2491895fc346d4cfad2f8, State: Running, Role: LEADER
I20260806 16:16:14.074388 6257 consensus_queue.cc:237] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } }
I20260806 16:16:14.077695 5760 catalog_manager.cc:5495] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2d02fa593a2491895fc346d4cfad2f8 (127.18.90.3). New cstate: current_term: 1 leader_uuid: "f2d02fa593a2491895fc346d4cfad2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d02fa593a2491895fc346d4cfad2f8" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 44109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20260806 16:16:14.106298 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.102651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.18.90.3:44109)
0806 16:16:14.102780 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:14.102783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:14.102796 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:16:14.102909 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:14.102911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:14.102965 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:14.103056 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:14.103330 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.103830 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.104709 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.104932 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.105044 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.105375 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:14.105826 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:14.105980 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:14.106039 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 16:16:14.106099 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:14.112442 6047 raft_consensus.cc:1261] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 1 FOLLOWER]: Refusing update from remote peer f2d02fa593a2491895fc346d4cfad2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:14.112442 5910 raft_consensus.cc:1261] T 2553b614cac74870ae319e2280b195e8 P 5c8d6f8f274847749163286bca28e908 [term 1 FOLLOWER]: Refusing update from remote peer f2d02fa593a2491895fc346d4cfad2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:14.113200 6259 consensus_queue.cc:1035] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8d6f8f274847749163286bca28e908" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:14.113454 6257 consensus_queue.cc:1035] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b5703f19c54f4d98d37b7ea88d9e09" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:14.130029 5760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:14.130226 5760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:14.132671 5760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baa7844fcef144769ef059ff0f6318ba: Sending DeleteTablet for 3 replicas of tablet 2553b614cac74870ae319e2280b195e8
I20260806 16:16:14.133491 6164 tablet_service.cc:1514] Processing DeleteTablet for tablet 2553b614cac74870ae319e2280b195e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:14 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:35906
I20260806 16:16:14.133495 5889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2553b614cac74870ae319e2280b195e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:14 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:43526
I20260806 16:16:14.133966 6026 tablet_service.cc:1514] Processing DeleteTablet for tablet 2553b614cac74870ae319e2280b195e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:14 UTC) from {username='kudu', principal='kudu/127.18.90.62@KRBTEST.COM'} at 127.0.0.1:43984
I20260806 16:16:14.134459 6273 tablet_replica.cc:330] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: stopping tablet replica
I20260806 16:16:14.134734 6271 tablet_replica.cc:330] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: stopping tablet replica
I20260806 16:16:14.134809 6273 raft_consensus.cc:2229] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:14.135000 6271 raft_consensus.cc:2229] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:16:14.135138 6273 raft_consensus.cc:2258] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:14.135142 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5825
I20260806 16:16:14.135319 6271 raft_consensus.cc:2258] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:14.136976 6271 ts_tablet_manager.cc:1905] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:14.139303 6271 ts_tablet_manager.cc:1918] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:14.139391 6271 log.cc:1196] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/2553b614cac74870ae319e2280b195e8
I20260806 16:16:14.139627 6271 ts_tablet_manager.cc:1939] T 2553b614cac74870ae319e2280b195e8 P f2d02fa593a2491895fc346d4cfad2f8: Deleting consensus metadata
I20260806 16:16:14.141865 5747 catalog_manager.cc:4841] TS f2d02fa593a2491895fc346d4cfad2f8 (127.18.90.3:44109): tablet 2553b614cac74870ae319e2280b195e8 (table test-table [id=cf7fa40dba224e2fbec3f3a8e9f18d68]) successfully deleted
I20260806 16:16:14.142802 6273 ts_tablet_manager.cc:1905] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:14.144984 6273 ts_tablet_manager.cc:1918] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:14.145102 6273 log.cc:1196] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/2553b614cac74870ae319e2280b195e8
I20260806 16:16:14.145393 6273 ts_tablet_manager.cc:1939] T 2553b614cac74870ae319e2280b195e8 P c6b5703f19c54f4d98d37b7ea88d9e09: Deleting consensus metadata
W20260806 16:16:14.146037 5748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:35475 (error 108)
I20260806 16:16:14.146231 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5963
I20260806 16:16:14.149233 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.148908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:35475 (local address 127.0.0.1:43532)
0806 16:16:14.149057 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:14.149139 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 16:16:14.149502 5748 catalog_manager.cc:4584] TS 5c8d6f8f274847749163286bca28e908 (127.18.90.1:35475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2553b614cac74870ae319e2280b195e8: Network error: Client connection negotiation failed: client connection to 127.18.90.1:35475: connect: Connection refused (error 111)
I20260806 16:16:14.152921 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6100
I20260806 16:16:14.154129 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.153008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:44025 (local address 127.0.0.1:43992)
0806 16:16:14.154008 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:14.154056 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.2:44025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":924}
I20260806 16:16:14.159722 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 5729
2026-08-06T16:16:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2469 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T16:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:14Z Disabled control of system clock
I20260806 16:16:14.192600 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:37975
--webserver_interface=127.18.90.62
--webserver_port=0
--builtin_ntp_servers=127.18.90.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:37975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:16:14.304973 6278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:14.305209 6278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:14.305243 6278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:14.309073 6278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:14.309118 6278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:14.309134 6278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:14.309157 6278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:14.309173 6278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:14.313715 6278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:37975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:14.314677 6278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:14.315953 6278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:14.324746 6283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:14.324784 6286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.324916 6278 server_base.cc:1034] running on GCE node
W20260806 16:16:14.324745 6284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.325605 6278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:14.326776 6278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:14.327970 6278 hybrid_clock.cc:648] HybridClock initialized: now 1786032974327952 us; error 35 us; skew 500 ppm
I20260806 16:16:14.329862 6278 webserver.cc:460] Webserver started at http://127.18.90.62:35111/ using document root <none> and password file <none>
I20260806 16:16:14.330452 6278 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:14.330544 6278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:14.330791 6278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:14.332964 6278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "0b08ba235dcb47bfb738560d786485e4"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "f174b10d93b06a628dbe05afbbf34c4c"
server_key_iv: "df4dfe4b961cf3754d3b3e225a31b090"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.333541 6278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "0b08ba235dcb47bfb738560d786485e4"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "f174b10d93b06a628dbe05afbbf34c4c"
server_key_iv: "df4dfe4b961cf3754d3b3e225a31b090"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.337268 6278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:16:14.339648 6292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.340917 6278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:16:14.341064 6278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "0b08ba235dcb47bfb738560d786485e4"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "f174b10d93b06a628dbe05afbbf34c4c"
server_key_iv: "df4dfe4b961cf3754d3b3e225a31b090"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.341173 6278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:14.359078 6278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:14.359714 6278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:14.359912 6278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:14.367233 6278 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:37975
I20260806 16:16:14.367242 6346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:37975 every 8 connection(s)
I20260806 16:16:14.368144 6278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:14.371121 6347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:14.376565 6347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4: Bootstrap starting.
I20260806 16:16:14.378367 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6278
I20260806 16:16:14.378510 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:14.378839 18792 external_mini_cluster.cc:1428] Setting key db5e9b27b99a4048a7942f8591d96666
I20260806 16:16:14.379367 6347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:14.380275 6347 log.cc:824] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:14.382323 6347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4: No bootstrap required, opened a new log
I20260806 16:16:14.386574 6347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b08ba235dcb47bfb738560d786485e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37975 } }
I20260806 16:16:14.386793 6347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:14.386833 6347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b08ba235dcb47bfb738560d786485e4, State: Initialized, Role: FOLLOWER
I20260806 16:16:14.387373 6347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b08ba235dcb47bfb738560d786485e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37975 } }
I20260806 16:16:14.387552 6347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:14.387635 6347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:14.387741 6347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:14.388468 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.380047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.18.90.62:37975)
0806 16:16:14.380729 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:14.380753 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:14.380787 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:16:14.381432 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:14.381448 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:14.381775 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:14.382096 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:14.382472 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.383545 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.384804 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.385249 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.385385 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.386348 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:14.386353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:14.386365 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:14.386749 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:14.386842 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:14.386873 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:14.387035 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:14.387242 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 16:16:14.387528 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
I20260806 16:16:14.388779 6347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b08ba235dcb47bfb738560d786485e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37975 } }
I20260806 16:16:14.389125 6347 leader_election.cc:302] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b08ba235dcb47bfb738560d786485e4; no voters:
I20260806 16:16:14.389413 6347 leader_election.cc:288] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:14.389544 6352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:14.389740 6352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b08ba235dcb47bfb738560d786485e4, State: Running, Role: LEADER
I20260806 16:16:14.390058 6352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b08ba235dcb47bfb738560d786485e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37975 } }
I20260806 16:16:14.390678 6347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:14.391214 6354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b08ba235dcb47bfb738560d786485e4. Latest consensus state: current_term: 1 leader_uuid: "0b08ba235dcb47bfb738560d786485e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b08ba235dcb47bfb738560d786485e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37975 } } }
I20260806 16:16:14.391353 6354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:14.391695 6357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:14.391669 6353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b08ba235dcb47bfb738560d786485e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b08ba235dcb47bfb738560d786485e4" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 37975 } } }
I20260806 16:16:14.391842 6353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:14.394938 6357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:14.400297 6357 catalog_manager.cc:1347] Generated new cluster ID: 3362fb3f737b4a518db32303f77c811c
I20260806 16:16:14.400372 6357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:14.410727 6357 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:14.411563 6357 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:14.415341 6357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4: Generated new TSK 0
I20260806 16:16:14.415954 6357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:14.422645 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.1:0
--local_ip_for_outbound_sockets=127.18.90.1
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37975
--builtin_ntp_servers=127.18.90.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:16:14.533309 6371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:14.533603 6371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:14.533675 6371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:14.537703 6371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:14.537792 6371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:14.537886 6371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.1
I20260806 16:16:14.542667 6371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.90.1
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.1
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:14.543879 6371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:14.545406 6371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:14.555024 6377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.555084 6371 server_base.cc:1034] running on GCE node
W20260806 16:16:14.555019 6376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:14.555019 6379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.555748 6371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:14.556439 6371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:14.557629 6371 hybrid_clock.cc:648] HybridClock initialized: now 1786032974557595 us; error 48 us; skew 500 ppm
I20260806 16:16:14.559495 6371 webserver.cc:460] Webserver started at http://127.18.90.1:36187/ using document root <none> and password file <none>
I20260806 16:16:14.560111 6371 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:14.560206 6371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:14.560474 6371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:14.562564 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data/instance:
uuid: "a13d311c9fdf4ab595457b852bf28936"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "88f54d0d0ec37c7d2aa8722429d98dc2"
server_key_iv: "b5fe5d1e4cb7a4387f18f9ea80d3bf82"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.563155 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance:
uuid: "a13d311c9fdf4ab595457b852bf28936"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "88f54d0d0ec37c7d2aa8722429d98dc2"
server_key_iv: "b5fe5d1e4cb7a4387f18f9ea80d3bf82"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.566929 6371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 16:16:14.569444 6386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.570657 6371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:14.570802 6371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal
uuid: "a13d311c9fdf4ab595457b852bf28936"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "88f54d0d0ec37c7d2aa8722429d98dc2"
server_key_iv: "b5fe5d1e4cb7a4387f18f9ea80d3bf82"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.570919 6371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:14.587935 6371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:14.588649 6371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:14.588860 6371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:14.589473 6371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:14.590454 6371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:14.590523 6371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.590592 6371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:14.590643 6371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.599701 6371 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.1:43673
I20260806 16:16:14.599752 6500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.1:43673 every 8 connection(s)
I20260806 16:16:14.600693 6371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/data/info.pb
I20260806 16:16:14.609900 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6371
I20260806 16:16:14.610059 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-0/wal/instance
I20260806 16:16:14.610275 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.602598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:58631 (local address 127.18.90.62:37975)
0806 16:16:14.602753 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:14.602757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:14.603811 (+ 1054us) server_negotiation.cc:407] Connection header received
0806 16:16:14.605307 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:14.605311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:14.605372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:14.605477 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:14.606417 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.606971 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.608135 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.608445 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.608564 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.609738 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:14.609741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:14.609744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:14.609759 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:14.609788 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:14.609796 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:14.609877 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:14.610045 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 16:16:14.610115 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 16:16:14.610425 18792 external_mini_cluster.cc:1428] Setting key a2df672724e956570082580e03f3a7e8
I20260806 16:16:14.611339 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.603013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:37975 (local address 127.18.90.1:58631)
0806 16:16:14.603649 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:14.603689 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:14.605072 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:14.605665 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:14.605671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:14.605762 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:14.606263 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.606276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.607102 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.607107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.608002 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.608011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.608613 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.608617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.608794 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.609463 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:14.609465 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:14.609507 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:14.609572 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:14.609880 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:14.609892 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:14.609901 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:16:14.610037 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 16:16:14.610320 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260806 16:16:14.612591 6501 heartbeater.cc:347] Connected to a master server at 127.18.90.62:37975
I20260806 16:16:14.612807 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.2:0
--local_ip_for_outbound_sockets=127.18.90.2
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37975
--builtin_ntp_servers=127.18.90.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:16:14.612835 6501 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:14.613370 6501 heartbeater.cc:511] Master 127.18.90.62:37975 requested a full tablet report, sending...
I20260806 16:16:14.614964 6310 ts_manager.cc:194] Registered new tserver with Master: a13d311c9fdf4ab595457b852bf28936 (127.18.90.1:43673)
I20260806 16:16:14.616098 6310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.1:58631
W20260806 16:16:14.724749 6504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:14.725033 6504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:14.725111 6504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:14.729377 6504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:14.729452 6504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:14.729537 6504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.2
I20260806 16:16:14.734336 6504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.90.2
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.2
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:14.735464 6504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:14.736891 6504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:14.746527 6512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:14.746567 6509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.746734 6504 server_base.cc:1034] running on GCE node
W20260806 16:16:14.746744 6510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.747290 6504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:14.747854 6504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:14.749010 6504 hybrid_clock.cc:648] HybridClock initialized: now 1786032974748982 us; error 39 us; skew 500 ppm
I20260806 16:16:14.750820 6504 webserver.cc:460] Webserver started at http://127.18.90.2:43037/ using document root <none> and password file <none>
I20260806 16:16:14.751379 6504 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:14.751438 6504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:14.751644 6504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:14.753684 6504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data/instance:
uuid: "f2c07e913223446db5536504b160aa7e"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "795b186b462fa00c9411ee8f7b7bed6e"
server_key_iv: "ed61dd43ae6b3cf4b4e400958dc13e26"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.754261 6504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance:
uuid: "f2c07e913223446db5536504b160aa7e"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "795b186b462fa00c9411ee8f7b7bed6e"
server_key_iv: "ed61dd43ae6b3cf4b4e400958dc13e26"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.757843 6504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:16:14.760198 6518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.761397 6504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:14.761615 6504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal
uuid: "f2c07e913223446db5536504b160aa7e"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "795b186b462fa00c9411ee8f7b7bed6e"
server_key_iv: "ed61dd43ae6b3cf4b4e400958dc13e26"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.761720 6504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:14.791407 6504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:14.792299 6504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:14.792519 6504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:14.793154 6504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:14.794121 6504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:14.794168 6504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.794237 6504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:14.794270 6504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.803053 6504 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.2:33757
I20260806 16:16:14.803084 6634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.2:33757 every 8 connection(s)
I20260806 16:16:14.803994 6504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/data/info.pb
I20260806 16:16:14.810192 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6504
I20260806 16:16:14.810333 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal/instance
I20260806 16:16:14.810680 18792 external_mini_cluster.cc:1428] Setting key 537132416c058a26be3bc4a55151c744
I20260806 16:16:14.812980 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.90.3:0
--local_ip_for_outbound_sockets=127.18.90.3
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37975
--builtin_ntp_servers=127.18.90.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:16:14.813760 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.805827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.2:40217 (local address 127.18.90.62:37975)
0806 16:16:14.806004 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:14.806009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:14.807074 (+ 1065us) server_negotiation.cc:407] Connection header received
0806 16:16:14.808458 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:14.808461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:14.808509 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:14.808589 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:14.809474 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.810003 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.811597 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.811890 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.812000 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.813311 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:14.813314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:14.813316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:14.813328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:14.813349 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:14.813356 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:14.813418 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:14.813600 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 16:16:14.813650 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 16:16:14.814980 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:14.806272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:37975 (local address 127.18.90.2:40217)
0806 16:16:14.806926 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:14.806967 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:14.808240 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:14.808750 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:14.808757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:14.808844 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:14.809352 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.809365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.810141 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.810145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.811470 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:14.811480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:14.812060 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:14.812064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:14.812304 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:14.812953 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:14.812957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:14.813010 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:14.813099 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:14.813444 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:14.813455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:14.813464 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:16:14.813593 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 16:16:14.813896 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260806 16:16:14.816311 6635 heartbeater.cc:347] Connected to a master server at 127.18.90.62:37975
I20260806 16:16:14.816536 6635 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:14.817040 6635 heartbeater.cc:511] Master 127.18.90.62:37975 requested a full tablet report, sending...
I20260806 16:16:14.818073 6310 ts_manager.cc:194] Registered new tserver with Master: f2c07e913223446db5536504b160aa7e (127.18.90.2:33757)
I20260806 16:16:14.818652 6310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.2:40217
W20260806 16:16:14.931595 6638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:14.931895 6638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:14.931938 6638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:14.936053 6638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:14.936110 6638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:14.936164 6638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.90.3
I20260806 16:16:14.941231 6638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.90.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.90.3
--webserver_port=0
--tserver_master_addrs=127.18.90.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.90.3
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:14.942427 6638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:14.943876 6638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:14.954052 6646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:14.954046 6644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:14.954183 6643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:14.954219 6638 server_base.cc:1034] running on GCE node
I20260806 16:16:14.954747 6638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:14.955408 6638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:14.956595 6638 hybrid_clock.cc:648] HybridClock initialized: now 1786032974956561 us; error 52 us; skew 500 ppm
I20260806 16:16:14.965559 6638 webserver.cc:460] Webserver started at http://127.18.90.3:44615/ using document root <none> and password file <none>
I20260806 16:16:14.966234 6638 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:14.966336 6638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:14.966558 6638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:14.968833 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data/instance:
uuid: "b7047099d4fc415c81a71faae64a8c8d"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "57a0206837508a5761d8018b48f2d77f"
server_key_iv: "305a5cec2afcff3c1c34ac1ced4fe91b"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.969424 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance:
uuid: "b7047099d4fc415c81a71faae64a8c8d"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "57a0206837508a5761d8018b48f2d77f"
server_key_iv: "305a5cec2afcff3c1c34ac1ced4fe91b"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.973227 6638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:16:14.975848 6653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.977214 6638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:16:14.977324 6638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal
uuid: "b7047099d4fc415c81a71faae64a8c8d"
format_stamp: "Formatted at 2026-08-06 16:16:14 on dist-test-slave-0b58"
server_key: "57a0206837508a5761d8018b48f2d77f"
server_key_iv: "305a5cec2afcff3c1c34ac1ced4fe91b"
server_key_version: "encryptionkey@0"
I20260806 16:16:14.977411 6638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:14.987107 6638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:14.987725 6638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:14.987870 6638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:14.988438 6638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:16:14.989326 6638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:16:14.989374 6638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.989413 6638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:16:14.989429 6638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:14.998574 6638 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.3:37779
I20260806 16:16:14.998602 6767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.3:37779 every 8 connection(s)
I20260806 16:16:14.999496 6638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/data/info.pb
I20260806 16:16:15.000679 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6638
I20260806 16:16:15.000808 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal/instance
I20260806 16:16:15.001106 18792 external_mini_cluster.cc:1428] Setting key 7d8a0a421d7aa07d4bf22ba162d8fd55
I20260806 16:16:15.009039 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.001562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.3:40787 (local address 127.18.90.62:37975)
0806 16:16:15.001717 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.001721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.003047 (+ 1326us) server_negotiation.cc:407] Connection header received
0806 16:16:15.004550 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.004553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.004599 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.004668 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:15.005553 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.006079 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.007206 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.007427 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.007503 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.008623 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:15.008626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:15.008628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:15.008640 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:15.008660 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:15.008668 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:15.008727 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.008867 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.008921 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:16:15.009925 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.002094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:37975 (local address 127.18.90.3:40787)
0806 16:16:15.002882 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.002939 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:15.004366 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:15.004830 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:15.004836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:15.004921 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:15.005435 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.005448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.006208 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.006211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.007098 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.007107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.007557 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.007560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.007700 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.008378 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:16:15.008380 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:15.008421 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:16:15.008485 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:15.008744 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:15.008750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:15.008754 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:16:15.008847 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 16:16:15.009066 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260806 16:16:15.011144 6768 heartbeater.cc:347] Connected to a master server at 127.18.90.62:37975
I20260806 16:16:15.011397 6768 heartbeater.cc:464] Registering TS with master...
I20260806 16:16:15.011927 6768 heartbeater.cc:511] Master 127.18.90.62:37975 requested a full tablet report, sending...
I20260806 16:16:15.013055 6310 ts_manager.cc:194] Registered new tserver with Master: b7047099d4fc415c81a71faae64a8c8d (127.18.90.3:37779)
I20260806 16:16:15.013664 6310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.90.3:40787
I20260806 16:16:15.015383 18792 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:16:15.021275 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.017852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.18.90.62:37975)
0806 16:16:15.017983 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.017986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.018177 (+ 191us) server_negotiation.cc:407] Connection header received
0806 16:16:15.018274 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.018276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.018314 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.018381 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:15.018604 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.019104 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.019884 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.020083 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.020157 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.020939 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:15.020942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:15.020944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:15.020955 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:15.020972 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:15.020980 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:15.021038 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.021112 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.021159 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:15.024089 6310 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:16:15.027215 6310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:16:15.042003 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.035630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:37779 (local address 127.0.0.1:59106)
0806 16:16:15.036087 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.036149 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:15.036316 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:15.037733 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:15.037737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:15.037757 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:15.037853 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.037859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.038888 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.038891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.040511 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.040522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.041599 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.041601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.041683 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.041714 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:16:15.041792 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 16:16:15.041882 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":111,"thread_start_us":87,"threads_started":1}
I20260806 16:16:15.042066 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.036501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.18.90.3:37779)
0806 16:16:15.037332 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.037343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.037363 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:16:15.037425 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.037429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.037604 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.037757 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:15.037970 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.038761 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.040645 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.041476 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.041593 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.041784 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.041840 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.041932 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":250,"threads_started":1}
I20260806 16:16:15.042630 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.036415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:33757 (local address 127.0.0.1:58924)
0806 16:16:15.036912 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.036925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:15.037017 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:15.037918 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:15.037923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:15.037943 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:15.038039 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.038044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.039411 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.039417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.040531 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.040542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.042224 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.042229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.042317 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.042339 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:15.042407 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 16:16:15.042501 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":79,"thread_start_us":85,"threads_started":1}
I20260806 16:16:15.043139 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.036838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.18.90.2:33757)
0806 16:16:15.037287 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.037298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.037331 (+ 33us) server_negotiation.cc:407] Connection header received
0806 16:16:15.037413 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.037418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.037604 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.037757 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:15.038180 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.039280 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.040688 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.041476 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.041593 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.041784 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.042863 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.042969 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":145,"threads_started":1}
I20260806 16:16:15.043838 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.035662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43673 (local address 127.0.0.1:55714)
0806 16:16:15.036558 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.036572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:15.036660 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:15.037511 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:15.037517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:15.037540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:15.037682 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.037690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.039144 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.039147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.040511 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.040522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.043429 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.043432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.043531 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.043553 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:16:15.043632 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:16:15.043700 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":132,"thread_start_us":114,"threads_started":1}
I20260806 16:16:15.045327 6701 tablet_service.cc:1467] Processing CreateTablet for tablet 9d48e3175ff54c62b6825c172ba99fa7 (DEFAULT_TABLE table=test-table [id=7c13d552477a4487a4e2e6fe3de83a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:15.045802 6566 tablet_service.cc:1467] Processing CreateTablet for tablet 9d48e3175ff54c62b6825c172ba99fa7 (DEFAULT_TABLE table=test-table [id=7c13d552477a4487a4e2e6fe3de83a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:15.046409 6701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d48e3175ff54c62b6825c172ba99fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:15.046792 6566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d48e3175ff54c62b6825c172ba99fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:15.051718 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.035863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.18.90.1:43673)
0806 16:16:15.037057 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.037072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.037095 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:16:15.037183 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.037193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.037357 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.037523 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:15.037831 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.039026 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.042265 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.043068 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.043184 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.043362 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.043686 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.043788 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":123,"threads_started":1}
I20260806 16:16:15.052659 6785 tablet_bootstrap.cc:492] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Bootstrap starting.
I20260806 16:16:15.053928 6786 tablet_bootstrap.cc:492] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Bootstrap starting.
I20260806 16:16:15.054841 6434 tablet_service.cc:1467] Processing CreateTablet for tablet 9d48e3175ff54c62b6825c172ba99fa7 (DEFAULT_TABLE table=test-table [id=7c13d552477a4487a4e2e6fe3de83a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:16:15.055835 6785 tablet_bootstrap.cc:654] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:15.055903 6434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d48e3175ff54c62b6825c172ba99fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:15.056030 6786 tablet_bootstrap.cc:654] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:15.056718 6785 log.cc:824] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:15.056859 6786 log.cc:824] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:15.058683 6785 tablet_bootstrap.cc:492] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: No bootstrap required, opened a new log
I20260806 16:16:15.058902 6785 ts_tablet_manager.cc:1397] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:16:15.059026 6786 tablet_bootstrap.cc:492] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: No bootstrap required, opened a new log
I20260806 16:16:15.059208 6786 ts_tablet_manager.cc:1397] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 16:16:15.062255 6785 raft_consensus.cc:348] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.062503 6785 raft_consensus.cc:374] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:15.062557 6785 raft_consensus.cc:729] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c07e913223446db5536504b160aa7e, State: Initialized, Role: FOLLOWER
I20260806 16:16:15.063019 6785 consensus_queue.cc:260] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.063534 6789 tablet_bootstrap.cc:492] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: Bootstrap starting.
I20260806 16:16:15.063447 6786 raft_consensus.cc:348] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.064113 6786 raft_consensus.cc:374] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:15.064100 6635 heartbeater.cc:503] Master 127.18.90.62:37975 was elected leader, sending a full tablet report...
I20260806 16:16:15.064198 6786 raft_consensus.cc:729] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7047099d4fc415c81a71faae64a8c8d, State: Initialized, Role: FOLLOWER
I20260806 16:16:15.064733 6786 consensus_queue.cc:260] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.064081 6785 ts_tablet_manager.cc:1428] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260806 16:16:15.065730 6768 heartbeater.cc:503] Master 127.18.90.62:37975 was elected leader, sending a full tablet report...
I20260806 16:16:15.065754 6786 ts_tablet_manager.cc:1428] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260806 16:16:15.066656 6789 tablet_bootstrap.cc:654] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:15.067631 6789 log.cc:824] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:15.069949 6789 tablet_bootstrap.cc:492] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: No bootstrap required, opened a new log
I20260806 16:16:15.070112 6789 ts_tablet_manager.cc:1397] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:16:15.073549 6789 raft_consensus.cc:348] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.073791 6789 raft_consensus.cc:374] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:15.073889 6789 raft_consensus.cc:729] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13d311c9fdf4ab595457b852bf28936, State: Initialized, Role: FOLLOWER
I20260806 16:16:15.074306 6789 consensus_queue.cc:260] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.074877 6501 heartbeater.cc:503] Master 127.18.90.62:37975 was elected leader, sending a full tablet report...
I20260806 16:16:15.075249 6789 ts_tablet_manager.cc:1428] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:16:15.085640 6793 raft_consensus.cc:482] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:16:15.085826 6793 raft_consensus.cc:504] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.086961 6793 leader_election.cc:288] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7047099d4fc415c81a71faae64a8c8d (127.18.90.3:37779), f2c07e913223446db5536504b160aa7e (127.18.90.2:33757)
I20260806 16:16:15.090924 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.087422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:36847 (local address 127.18.90.2:33757)
0806 16:16:15.087539 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.087544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.087559 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:16:15.087617 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.087620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.087674 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.087770 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:15.088115 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.088739 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.089803 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.090188 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.090307 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.090374 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.090703 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.090782 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:16:15.090929 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.087326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.90.1:33389 (local address 127.18.90.3:37779)
0806 16:16:15.087470 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.087475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.087794 (+ 319us) server_negotiation.cc:407] Connection header received
0806 16:16:15.087909 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.087912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.087965 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.088050 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:16:15.088370 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.088845 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.089870 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.090266 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.090392 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.090453 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.090720 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.090800 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:16:15.090924 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.087188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.2:33757 (local address 127.18.90.1:36847)
0806 16:16:15.087324 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.087342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:15.087432 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:15.087813 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:15.087819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:15.087859 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:15.087986 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.087998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.088858 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.088864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.089667 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.089676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.090321 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.090325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.090549 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.090591 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 16:16:15.090674 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 16:16:15.090793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":63}
I20260806 16:16:15.090950 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.087188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.3:37779 (local address 127.18.90.1:33389)
0806 16:16:15.087690 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.087705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:15.087800 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:15.088088 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:15.088094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:15.088112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:16:15.088228 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.088234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.088959 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.088966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.089733 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:15.089749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.090434 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.090438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:15.090561 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.090605 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 16:16:15.090690 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:16:15.090803 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":165,"thread_start_us":160,"threads_started":1}
I20260806 16:16:15.091607 6722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d48e3175ff54c62b6825c172ba99fa7" candidate_uuid: "a13d311c9fdf4ab595457b852bf28936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7047099d4fc415c81a71faae64a8c8d" is_pre_election: true
I20260806 16:16:15.091934 6586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d48e3175ff54c62b6825c172ba99fa7" candidate_uuid: "a13d311c9fdf4ab595457b852bf28936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c07e913223446db5536504b160aa7e" is_pre_election: true
I20260806 16:16:15.091979 6722 raft_consensus.cc:2454] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13d311c9fdf4ab595457b852bf28936 in term 0.
I20260806 16:16:15.092227 6586 raft_consensus.cc:2454] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13d311c9fdf4ab595457b852bf28936 in term 0.
I20260806 16:16:15.092530 6390 leader_election.cc:302] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13d311c9fdf4ab595457b852bf28936, b7047099d4fc415c81a71faae64a8c8d; no voters:
I20260806 16:16:15.092839 6793 raft_consensus.cc:2781] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:16:15.092965 6793 raft_consensus.cc:482] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:16:15.093024 6793 raft_consensus.cc:3037] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:15.094091 6793 raft_consensus.cc:504] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.094497 6793 leader_election.cc:288] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [CANDIDATE]: Term 1 election: Requested vote from peers b7047099d4fc415c81a71faae64a8c8d (127.18.90.3:37779), f2c07e913223446db5536504b160aa7e (127.18.90.2:33757)
I20260806 16:16:15.094832 6722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d48e3175ff54c62b6825c172ba99fa7" candidate_uuid: "a13d311c9fdf4ab595457b852bf28936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7047099d4fc415c81a71faae64a8c8d"
I20260806 16:16:15.094873 6586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d48e3175ff54c62b6825c172ba99fa7" candidate_uuid: "a13d311c9fdf4ab595457b852bf28936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c07e913223446db5536504b160aa7e"
I20260806 16:16:15.094975 6586 raft_consensus.cc:3037] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:15.094992 6722 raft_consensus.cc:3037] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:15.096058 6586 raft_consensus.cc:2454] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13d311c9fdf4ab595457b852bf28936 in term 1.
I20260806 16:16:15.096206 6722 raft_consensus.cc:2454] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13d311c9fdf4ab595457b852bf28936 in term 1.
I20260806 16:16:15.096459 6389 leader_election.cc:302] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a13d311c9fdf4ab595457b852bf28936, f2c07e913223446db5536504b160aa7e; no voters:
I20260806 16:16:15.096642 6793 raft_consensus.cc:2781] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:15.096866 6793 raft_consensus.cc:686] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [term 1 LEADER]: Becoming Leader. State: Replica: a13d311c9fdf4ab595457b852bf28936, State: Running, Role: LEADER
I20260806 16:16:15.097237 6793 consensus_queue.cc:237] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } }
I20260806 16:16:15.100430 6310 catalog_manager.cc:5495] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 reported cstate change: term changed from 0 to 1, leader changed from <none> to a13d311c9fdf4ab595457b852bf28936 (127.18.90.1). New cstate: current_term: 1 leader_uuid: "a13d311c9fdf4ab595457b852bf28936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13d311c9fdf4ab595457b852bf28936" member_type: VOTER last_known_addr { host: "127.18.90.1" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 } health_report { overall_health: UNKNOWN } } }
W20260806 16:16:15.101961 6502 tablet.cc:2369] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:16:15.120790 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.117350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.18.90.1:43673)
0806 16:16:15.117486 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:15.117490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:15.117508 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:16:15.117587 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:15.117590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:15.117647 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:15.117738 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:15.117978 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.118486 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.119294 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:15.119500 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:15.119576 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:15.119850 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:15.120315 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:15.120405 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:15.120515 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 16:16:15.120563 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:15.127019 6722 raft_consensus.cc:1261] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 1 FOLLOWER]: Refusing update from remote peer a13d311c9fdf4ab595457b852bf28936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:15.127017 6586 raft_consensus.cc:1261] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 1 FOLLOWER]: Refusing update from remote peer a13d311c9fdf4ab595457b852bf28936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:15.127738 6793 consensus_queue.cc:1035] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c07e913223446db5536504b160aa7e" member_type: VOTER last_known_addr { host: "127.18.90.2" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:15.127949 6795 consensus_queue.cc:1035] T 9d48e3175ff54c62b6825c172ba99fa7 P a13d311c9fdf4ab595457b852bf28936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7047099d4fc415c81a71faae64a8c8d" member_type: VOTER last_known_addr { host: "127.18.90.3" port: 37779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:15.136838 6800 mvcc.cc:205] Tried to move back new op lower bound from 7315591066112270336 to 7315591066001252352. Current Snapshot: MvccSnapshot[applied={T|T < 7315591066112270336}]
I20260806 16:16:15.145800 6307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:15.145960 6307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:16:15.148398 6307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b08ba235dcb47bfb738560d786485e4: Sending DeleteTablet for 3 replicas of tablet 9d48e3175ff54c62b6825c172ba99fa7
I20260806 16:16:15.149209 6701 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d48e3175ff54c62b6825c172ba99fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:15 UTC) from {username='slave'} at 127.0.0.1:59106
I20260806 16:16:15.149503 6434 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d48e3175ff54c62b6825c172ba99fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:15 UTC) from {username='slave'} at 127.0.0.1:55714
I20260806 16:16:15.149888 6809 tablet_replica.cc:330] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: stopping tablet replica
I20260806 16:16:15.149971 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6371
I20260806 16:16:15.150019 6566 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d48e3175ff54c62b6825c172ba99fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:16:15 UTC) from {username='slave'} at 127.0.0.1:58924
I20260806 16:16:15.150967 6811 tablet_replica.cc:330] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: stopping tablet replica
I20260806 16:16:15.150974 6809 raft_consensus.cc:2229] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:15.151269 6811 raft_consensus.cc:2229] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:16:15.151350 6809 raft_consensus.cc:2258] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:15.151599 6811 raft_consensus.cc:2258] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:16:15.153307 6811 ts_tablet_manager.cc:1905] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:15.154017 6809 ts_tablet_manager.cc:1905] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:16:15.156481 6809 ts_tablet_manager.cc:1918] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:15.156601 6809 log.cc:1196] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-2/wal/wals/9d48e3175ff54c62b6825c172ba99fa7
I20260806 16:16:15.156888 6809 ts_tablet_manager.cc:1939] T 9d48e3175ff54c62b6825c172ba99fa7 P b7047099d4fc415c81a71faae64a8c8d: Deleting consensus metadata
I20260806 16:16:15.156985 6811 ts_tablet_manager.cc:1918] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:16:15.157096 6811 log.cc:1196] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Deleting WAL directory at /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032720994684-18792-0/minicluster-data/ts-1/wal/wals/9d48e3175ff54c62b6825c172ba99fa7
I20260806 16:16:15.157425 6811 ts_tablet_manager.cc:1939] T 9d48e3175ff54c62b6825c172ba99fa7 P f2c07e913223446db5536504b160aa7e: Deleting consensus metadata
I20260806 16:16:15.158288 6296 catalog_manager.cc:4841] TS b7047099d4fc415c81a71faae64a8c8d (127.18.90.3:37779): tablet 9d48e3175ff54c62b6825c172ba99fa7 (table test-table [id=7c13d552477a4487a4e2e6fe3de83a0f]) successfully deleted
I20260806 16:16:15.158674 6295 catalog_manager.cc:4841] TS f2c07e913223446db5536504b160aa7e (127.18.90.2:33757): tablet 9d48e3175ff54c62b6825c172ba99fa7 (table test-table [id=7c13d552477a4487a4e2e6fe3de83a0f]) successfully deleted
W20260806 16:16:15.159879 6295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.90.1:43673 (error 108)
I20260806 16:16:15.160754 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6504
I20260806 16:16:15.161006 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:15.160686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.1:43673 (local address 127.0.0.1:55730)
0806 16:16:15.160831 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:15.160908 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.1:43673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 16:16:15.161878 6295 catalog_manager.cc:4584] TS a13d311c9fdf4ab595457b852bf28936 (127.18.90.1:43673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d48e3175ff54c62b6825c172ba99fa7: Network error: Client connection negotiation failed: client connection to 127.18.90.1:43673: connect: Connection refused (error 111)
I20260806 16:16:15.167179 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6638
I20260806 16:16:15.173451 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6278
2026-08-06T16:16:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1012 ms)
[----------] 6 tests from EncryptionPolicyTest (10442 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:15 dist-test-slave-0b58 krb5kdc[6815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:15 dist-test-slave-0b58 krb5kdc[6815](info): set up 2 sockets
Aug 06 16:16:15 dist-test-slave-0b58 krb5kdc[6815](info): commencing operation
krb5kdc: starting...
W20260806 16:16:16.208621 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:16 dist-test-slave-0b58 krb5kdc[6815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:16.339200 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:16.450260 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:16.450502 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:16.450541 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:16.454567 6828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:16.454623 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:16.454643 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:16.454658 6828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:16.454672 6828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:16.459812 6828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:16.460877 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:16.461889 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:16.470381 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:16.470355 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:16.470399 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:16.470705 6828 server_base.cc:1034] running on GCE node
I20260806 16:16:16.471328 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:16.472514 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:16.473702 6828 hybrid_clock.cc:648] HybridClock initialized: now 1786032976473682 us; error 29 us; skew 500 ppm
Aug 06 16:16:16 dist-test-slave-0b58 krb5kdc[6815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:16.476457 6828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:16.477427 6828 webserver.cc:460] Webserver started at http://127.0.0.1:35781/ using document root <none> and password file <none>
I20260806 16:16:16.478008 6828 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:16.478106 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:16.478344 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:16.480515 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "6c0702815a474698a186d84ae88a95a0"
format_stamp: "Formatted at 2026-08-06 16:16:16 on dist-test-slave-0b58"
server_key: "1d16af6b7e20b6c76c855c02180a89b9"
server_key_iv: "76086205d4467e43cc29cb2e5d60ccf8"
server_key_version: "encryptionkey@0"
I20260806 16:16:16.481088 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "6c0702815a474698a186d84ae88a95a0"
format_stamp: "Formatted at 2026-08-06 16:16:16 on dist-test-slave-0b58"
server_key: "1d16af6b7e20b6c76c855c02180a89b9"
server_key_iv: "76086205d4467e43cc29cb2e5d60ccf8"
server_key_version: "encryptionkey@0"
I20260806 16:16:16.484721 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:16.487085 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:16.488376 6828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:16.488535 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "6c0702815a474698a186d84ae88a95a0"
format_stamp: "Formatted at 2026-08-06 16:16:16 on dist-test-slave-0b58"
server_key: "1d16af6b7e20b6c76c855c02180a89b9"
server_key_iv: "76086205d4467e43cc29cb2e5d60ccf8"
server_key_version: "encryptionkey@0"
I20260806 16:16:16.488642 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:16.502108 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:16.504910 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:16.505118 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:16.512252 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40887
I20260806 16:16:16.512311 6897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40887 every 8 connection(s)
I20260806 16:16:16.513186 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:16.515889 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6828
I20260806 16:16:16.515920 6898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:16.516028 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:16.516407 18792 external_mini_cluster.cc:1428] Setting key 373c8541540a9ced46af76283220a393
I20260806 16:16:16.523368 6898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0: Bootstrap starting.
I20260806 16:16:16.526001 6898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0: Neither blocks nor log segments found. Creating new log.
Aug 06 16:16:16 dist-test-slave-0b58 krb5kdc[6815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:16:16.526968 6898 log.cc:824] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:16.529330 6898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0: No bootstrap required, opened a new log
I20260806 16:16:16.532363 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:16.517926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38517 (local address 127.0.0.1:40887)
0806 16:16:16.518383 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:16.518403 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:16.518449 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:16:16.519496 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:16.519522 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:16.519972 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:16.520386 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:16.521102 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:16.522625 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:16.523493 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:16.524179 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:16.524385 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:16.527300 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:16.527305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:16.527322 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:16.527351 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:16.529314 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:16.529942 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:16.529947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:16.529951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:16.530060 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:16.530422 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:16.530424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:16.530426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:16.530802 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:16.530966 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:16.531295 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 16:16:16.531533 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260806 16:16:16.533613 6898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0702815a474698a186d84ae88a95a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260806 16:16:16.533839 6898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:16.533905 6898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0702815a474698a186d84ae88a95a0, State: Initialized, Role: FOLLOWER
I20260806 16:16:16.534359 6898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0702815a474698a186d84ae88a95a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260806 16:16:16.534502 6898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:16.534586 6898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:16.534694 6898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:16.535451 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:16.535825 6898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0702815a474698a186d84ae88a95a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260806 16:16:16.536164 6898 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c0702815a474698a186d84ae88a95a0; no voters:
I20260806 16:16:16.536509 6898 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:16.536610 6903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:16.536787 6903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0702815a474698a186d84ae88a95a0, State: Running, Role: LEADER
I20260806 16:16:16.537073 6903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0702815a474698a186d84ae88a95a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260806 16:16:16.537799 6898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:16.538235 6903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0702815a474698a186d84ae88a95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0702815a474698a186d84ae88a95a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } } }
I20260806 16:16:16.538350 6903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:16.538539 6903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0702815a474698a186d84ae88a95a0. Latest consensus state: current_term: 1 leader_uuid: "6c0702815a474698a186d84ae88a95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0702815a474698a186d84ae88a95a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } } }
I20260806 16:16:16.538628 6903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:16.540443 6913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:16.543533 6913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:16.546429 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:16.537651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52431 (local address 127.0.0.1:40887)
0806 16:16:16.537766 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:16.537770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:16.538035 (+ 265us) server_negotiation.cc:407] Connection header received
0806 16:16:16.538169 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:16.538172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:16.538221 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:16.539274 (+ 1053us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:16.539396 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:16.540122 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:16.541172 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:16.541396 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:16.541506 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:16.542927 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:16.542931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:16.542933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:16.542958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:16.544609 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:16.545103 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:16.545107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:16.545108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:16.545194 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:16.545535 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:16.545539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:16.545540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:16.545722 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:16.545879 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:16.546168 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 16:16:16.546271 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:16:16.549451 6913 catalog_manager.cc:1347] Generated new cluster ID: 02b7cbf17f5047dfb366bfac7ea274ed
I20260806 16:16:16.549526 6913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:16.564821 6913 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:16.565932 6913 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:16.571529 6913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c0702815a474698a186d84ae88a95a0: Generated new TSK 0
I20260806 16:16:16.572307 6913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:16.582307 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6828
W20260806 16:16:16.589051 6813 connection.cc:535] client connection to 127.0.0.1:40887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40887): Connection reset by peer (error 104)
2026-08-06T16:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:16 dist-test-slave-0b58 krb5kdc[6930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:16 dist-test-slave-0b58 krb5kdc[6930](info): set up 2 sockets
Aug 06 16:16:16 dist-test-slave-0b58 krb5kdc[6930](info): commencing operation
krb5kdc: starting...
W20260806 16:16:17.620836 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:17 dist-test-slave-0b58 krb5kdc[6930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:17.747622 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:17.863199 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:17.863477 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:17.863521 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:17.867560 6943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:17.867625 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:17.867645 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:17.867661 6943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:17.867676 6943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:17.873015 6943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:17.874063 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:17.875053 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:17.883522 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:17.883634 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:17.883652 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:17.883744 6943 server_base.cc:1034] running on GCE node
I20260806 16:16:17.884543 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:17.885599 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:17.886762 6943 hybrid_clock.cc:648] HybridClock initialized: now 1786032977886733 us; error 42 us; skew 500 ppm
Aug 06 16:16:17 dist-test-slave-0b58 krb5kdc[6930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:17.889624 6943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:17.890460 6943 webserver.cc:460] Webserver started at http://127.0.0.1:45471/ using document root <none> and password file <none>
I20260806 16:16:17.890985 6943 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:17.891042 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:17.891203 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:17.893339 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "c2796c8ebbb94249acc8fb82ea6ac393"
format_stamp: "Formatted at 2026-08-06 16:16:17 on dist-test-slave-0b58"
server_key: "c894ad021b1166f80ecf5bd86c441cdd"
server_key_iv: "8c7e67d69c47858ef82ae3f7b8b9ffe3"
server_key_version: "encryptionkey@0"
I20260806 16:16:17.893925 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "c2796c8ebbb94249acc8fb82ea6ac393"
format_stamp: "Formatted at 2026-08-06 16:16:17 on dist-test-slave-0b58"
server_key: "c894ad021b1166f80ecf5bd86c441cdd"
server_key_iv: "8c7e67d69c47858ef82ae3f7b8b9ffe3"
server_key_version: "encryptionkey@0"
I20260806 16:16:17.897853 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:16:17.900502 6958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:17.901796 6943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:16:17.901944 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "c2796c8ebbb94249acc8fb82ea6ac393"
format_stamp: "Formatted at 2026-08-06 16:16:17 on dist-test-slave-0b58"
server_key: "c894ad021b1166f80ecf5bd86c441cdd"
server_key_iv: "8c7e67d69c47858ef82ae3f7b8b9ffe3"
server_key_version: "encryptionkey@0"
I20260806 16:16:17.902062 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:17.919384 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:17.922881 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:17.923099 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:17.930486 6943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42877
I20260806 16:16:17.930507 7014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42877 every 8 connection(s)
I20260806 16:16:17.931499 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:17.934310 7015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:17.934535 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 6943
I20260806 16:16:17.934686 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:17.935024 18792 external_mini_cluster.cc:1428] Setting key e2be8728313b4cd224e571f2466e36f7
I20260806 16:16:17.941311 7015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393: Bootstrap starting.
I20260806 16:16:17.943802 7015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:17.944084 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:17.936525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59267 (local address 127.0.0.1:42877)
0806 16:16:17.936956 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:17.936972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:17.937008 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:16:17.937734 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:17.937752 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:17.938079 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:17.938373 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:17.938718 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:17.939656 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:17.940853 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:17.941310 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:17.941461 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:17.942333 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:17.942338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:17.942350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:17.942729 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:17.942780 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:17.942802 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:17.942900 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:17.943106 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 16:16:17.943307 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260806 16:16:17.944597 7015 log.cc:824] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:17.946867 7015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393: No bootstrap required, opened a new log
I20260806 16:16:17.947248 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:17.951687 7015 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260806 16:16:17.952008 7015 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:17.952126 7015 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2796c8ebbb94249acc8fb82ea6ac393, State: Initialized, Role: FOLLOWER
I20260806 16:16:17.952669 7015 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260806 16:16:17.952849 7015 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:17.952934 7015 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:17.953027 7015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:17.953967 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:17.949682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50717 (local address 127.0.0.1:42877)
0806 16:16:17.949808 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:17.949814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:17.949967 (+ 153us) server_negotiation.cc:407] Connection header received
0806 16:16:17.950078 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:17.950080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:17.950117 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:17.950183 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:17.950480 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:17.950966 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:17.952097 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:17.952461 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:17.952570 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:17.953434 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:17.953437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:17.953438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:17.953450 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:17.953473 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:17.953483 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:17.953554 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:17.953752 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 16:16:17.953815 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:16:17.954299 7015 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260806 16:16:17.954670 7015 leader_election.cc:302] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2796c8ebbb94249acc8fb82ea6ac393; no voters:
I20260806 16:16:17.955011 7015 leader_election.cc:288] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:17.955123 7025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:17.955374 7025 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [term 1 LEADER]: Becoming Leader. State: Replica: c2796c8ebbb94249acc8fb82ea6ac393, State: Running, Role: LEADER
I20260806 16:16:17.955668 7025 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260806 16:16:17.956306 7015 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:17.956878 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2796c8ebbb94249acc8fb82ea6ac393. Latest consensus state: current_term: 1 leader_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } } }
I20260806 16:16:17.957001 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:17.957412 7031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:17.957722 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2796c8ebbb94249acc8fb82ea6ac393" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } } }
I20260806 16:16:17.957894 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:17.960678 7031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:17.966589 7031 catalog_manager.cc:1347] Generated new cluster ID: c7efe64965d546f7bdf5a5ed4c0831ea
I20260806 16:16:17.966681 7031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:17.977551 7031 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:17.978794 7031 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:17.983553 7031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2796c8ebbb94249acc8fb82ea6ac393: Generated new TSK 0
I20260806 16:16:17.984455 7031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:17.997361 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 6943
W20260806 16:16:18.003899 6928 connection.cc:535] client connection to 127.0.0.1:42877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42877): Connection reset by peer (error 104)
2026-08-06T16:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:18 dist-test-slave-0b58 krb5kdc[7047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:18 dist-test-slave-0b58 krb5kdc[7047](info): set up 2 sockets
Aug 06 16:16:18 dist-test-slave-0b58 krb5kdc[7047](info): commencing operation
krb5kdc: starting...
W20260806 16:16:19.024701 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:19 dist-test-slave-0b58 krb5kdc[7047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:19.149637 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:19.263085 7061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:19.263402 7061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:19.263481 7061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:19.267480 7061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:19.267576 7061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:19.267616 7061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:19.267668 7061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:19.267709 7061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:19.273037 7061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:19.274152 7061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:19.275183 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:19.283818 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:19.283808 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:19.283808 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:19.283864 7061 server_base.cc:1034] running on GCE node
I20260806 16:16:19.284734 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:19.285820 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:19.286985 7061 hybrid_clock.cc:648] HybridClock initialized: now 1786032979286941 us; error 54 us; skew 500 ppm
Aug 06 16:16:19 dist-test-slave-0b58 krb5kdc[7047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:19.289791 7061 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:19.290761 7061 webserver.cc:460] Webserver started at http://127.0.0.1:40793/ using document root <none> and password file <none>
I20260806 16:16:19.291335 7061 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:19.291396 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:19.291633 7061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:19.293751 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "94440971f2fd478b99cb2ef2b626c668"
format_stamp: "Formatted at 2026-08-06 16:16:19 on dist-test-slave-0b58"
server_key: "01a38299c1462e874e9677786c66b8a4"
server_key_iv: "38ab71138b3d539892ce6fbd58a53a5b"
server_key_version: "encryptionkey@0"
I20260806 16:16:19.294325 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "94440971f2fd478b99cb2ef2b626c668"
format_stamp: "Formatted at 2026-08-06 16:16:19 on dist-test-slave-0b58"
server_key: "01a38299c1462e874e9677786c66b8a4"
server_key_iv: "38ab71138b3d539892ce6fbd58a53a5b"
server_key_version: "encryptionkey@0"
I20260806 16:16:19.298221 7061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:16:19.300664 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:19.301944 7061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:19.302100 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "94440971f2fd478b99cb2ef2b626c668"
format_stamp: "Formatted at 2026-08-06 16:16:19 on dist-test-slave-0b58"
server_key: "01a38299c1462e874e9677786c66b8a4"
server_key_iv: "38ab71138b3d539892ce6fbd58a53a5b"
server_key_version: "encryptionkey@0"
I20260806 16:16:19.302239 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:19.306988 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:19.309994 7061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:19.310218 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:19.317762 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44459
I20260806 16:16:19.317790 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44459 every 8 connection(s)
I20260806 16:16:19.318778 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:19.321718 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:19.327026 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7061
I20260806 16:16:19.327201 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:19.327515 18792 external_mini_cluster.cc:1428] Setting key 2b89a8b3eb6c04ad64bc5d52464c928e
I20260806 16:16:19.327796 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668: Bootstrap starting.
I20260806 16:16:19.330448 7131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:19.331274 7131 log.cc:824] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:19.333017 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:19.328956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49637 (local address 127.0.0.1:44459)
0806 16:16:19.329453 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:19.329468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:19.329505 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:16:19.330160 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:19.330177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:19.330468 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:19.330863 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:19.331736 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:19.331741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:19.331752 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:19.331762 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:19.331818 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:19.331837 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:19.331956 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:19.332069 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 16:16:19.332243 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":147,"threads_started":1}
I20260806 16:16:19.333361 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668: No bootstrap required, opened a new log
I20260806 16:16:19.335986 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:19.337599 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94440971f2fd478b99cb2ef2b626c668" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260806 16:16:19.337845 7131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:19.337908 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94440971f2fd478b99cb2ef2b626c668, State: Initialized, Role: FOLLOWER
I20260806 16:16:19.338404 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94440971f2fd478b99cb2ef2b626c668" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260806 16:16:19.338581 7131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:19.338653 7131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:19.338783 7131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:19.340101 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:19.338162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56497 (local address 127.0.0.1:44459)
0806 16:16:19.338285 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:19.338289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:19.338490 (+ 201us) server_negotiation.cc:407] Connection header received
0806 16:16:19.338609 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:19.338612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:19.338652 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:19.338738 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:19.339657 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:19.339660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:19.339663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:19.339674 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:19.339703 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:19.339715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:19.339817 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:19.339910 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:16:19.339951 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:16:19.340090 7131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94440971f2fd478b99cb2ef2b626c668" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260806 16:16:19.340517 7131 leader_election.cc:302] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94440971f2fd478b99cb2ef2b626c668; no voters:
I20260806 16:16:19.340840 7131 leader_election.cc:288] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:19.340956 7141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:19.341245 7141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [term 1 LEADER]: Becoming Leader. State: Replica: 94440971f2fd478b99cb2ef2b626c668, State: Running, Role: LEADER
I20260806 16:16:19.341545 7141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94440971f2fd478b99cb2ef2b626c668" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260806 16:16:19.342082 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:19.342610 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94440971f2fd478b99cb2ef2b626c668. Latest consensus state: current_term: 1 leader_uuid: "94440971f2fd478b99cb2ef2b626c668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94440971f2fd478b99cb2ef2b626c668" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } } }
I20260806 16:16:19.342800 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:19.343080 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94440971f2fd478b99cb2ef2b626c668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94440971f2fd478b99cb2ef2b626c668" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } } }
I20260806 16:16:19.343230 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:19.343269 7149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:19.346302 7149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:19.351887 7149 catalog_manager.cc:1347] Generated new cluster ID: 52fcd29898f947b78b90f72cd345c939
I20260806 16:16:19.351959 7149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:19.361202 7149 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:19.361533 7149 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:19.373211 7149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94440971f2fd478b99cb2ef2b626c668: Generated new TSK 0
I20260806 16:16:19.373942 7149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:19.382194 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7061
2026-08-06T16:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:19 dist-test-slave-0b58 krb5kdc[7164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:19 dist-test-slave-0b58 krb5kdc[7164](info): set up 2 sockets
Aug 06 16:16:19 dist-test-slave-0b58 krb5kdc[7164](info): commencing operation
krb5kdc: starting...
W20260806 16:16:20.424798 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:20 dist-test-slave-0b58 krb5kdc[7164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:20.563251 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:20.675472 7177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:20.675797 7177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:20.675890 7177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:20.679909 7177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:20.680011 7177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:20.680048 7177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:20.680110 7177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:20.680147 7177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:20.685474 7177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:20.686609 7177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:20.687647 7177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:16:20.696358 7177 server_base.cc:1034] running on GCE node
W20260806 16:16:20.696221 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:20.696220 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:20.696244 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:20.697110 7177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:20.698215 7177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:20.699404 7177 hybrid_clock.cc:648] HybridClock initialized: now 1786032980699384 us; error 30 us; skew 500 ppm
Aug 06 16:16:20 dist-test-slave-0b58 krb5kdc[7164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:20.702046 7177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:20.702872 7177 webserver.cc:460] Webserver started at http://127.0.0.1:34333/ using document root <none> and password file <none>
I20260806 16:16:20.703454 7177 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:20.703542 7177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:20.703780 7177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:20.705873 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "da0067483d1a4a17832be0d902550788"
format_stamp: "Formatted at 2026-08-06 16:16:20 on dist-test-slave-0b58"
server_key: "8374935a1f5b1f3ce908959c0108aad2"
server_key_iv: "76b4a6b94b3e09bef9e4a4d0e5794622"
server_key_version: "encryptionkey@0"
I20260806 16:16:20.706446 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "da0067483d1a4a17832be0d902550788"
format_stamp: "Formatted at 2026-08-06 16:16:20 on dist-test-slave-0b58"
server_key: "8374935a1f5b1f3ce908959c0108aad2"
server_key_iv: "76b4a6b94b3e09bef9e4a4d0e5794622"
server_key_version: "encryptionkey@0"
I20260806 16:16:20.710084 7177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:16:20.712396 7193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:20.713577 7177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:20.713739 7177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "da0067483d1a4a17832be0d902550788"
format_stamp: "Formatted at 2026-08-06 16:16:20 on dist-test-slave-0b58"
server_key: "8374935a1f5b1f3ce908959c0108aad2"
server_key_iv: "76b4a6b94b3e09bef9e4a4d0e5794622"
server_key_version: "encryptionkey@0"
I20260806 16:16:20.713850 7177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:20.730707 7177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:20.733525 7177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:20.733731 7177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:20.740588 7177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40615
I20260806 16:16:20.740631 7246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40615 every 8 connection(s)
I20260806 16:16:20.741565 7177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:20.744393 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:20.749796 7247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788: Bootstrap starting.
I20260806 16:16:20.750064 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7177
I20260806 16:16:20.750231 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:20.750510 18792 external_mini_cluster.cc:1428] Setting key a95eb97035713516c322bfb62b2280f8
I20260806 16:16:20.752405 7247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:20.753162 7247 log.cc:824] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:20.754978 7247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788: No bootstrap required, opened a new log
I20260806 16:16:20.758296 7247 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0067483d1a4a17832be0d902550788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20260806 16:16:20.758534 7247 raft_consensus.cc:374] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:20.758615 7247 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0067483d1a4a17832be0d902550788, State: Initialized, Role: FOLLOWER
I20260806 16:16:20.759122 7247 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0067483d1a4a17832be0d902550788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
Aug 06 16:16:20 dist-test-slave-0b58 krb5kdc[7164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:16:20.759279 7247 raft_consensus.cc:388] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:20.759351 7247 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:20.759460 7247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:20.760564 7247 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0067483d1a4a17832be0d902550788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20260806 16:16:20.760962 7247 leader_election.cc:302] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da0067483d1a4a17832be0d902550788; no voters:
I20260806 16:16:20.761349 7247 leader_election.cc:288] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:20.761400 7252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:20.761577 7252 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [term 1 LEADER]: Becoming Leader. State: Replica: da0067483d1a4a17832be0d902550788, State: Running, Role: LEADER
I20260806 16:16:20.761883 7252 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0067483d1a4a17832be0d902550788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } }
I20260806 16:16:20.762573 7247 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:20.762979 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da0067483d1a4a17832be0d902550788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0067483d1a4a17832be0d902550788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } } }
I20260806 16:16:20.763080 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:20.763515 7256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:20.765913 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:20.751775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60207 (local address 127.0.0.1:40615)
0806 16:16:20.752185 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:20.752201 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:20.752237 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:16:20.753008 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:20.753026 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:20.753404 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:20.753744 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:20.754553 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:20.755576 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:20.756505 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:20.756997 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:20.757164 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:20.759961 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:20.759966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:20.759977 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:20.760003 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:20.762089 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:20.762589 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:20.762594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:20.762600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:20.762748 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:20.763055 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:20.763059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:20.763060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:20.763620 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:20.763820 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:20.764658 (+ 838us) server_negotiation.cc:299] Negotiation successful
0806 16:16:20.764963 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":131,"threads_started":1}
I20260806 16:16:20.766767 7256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:20.769351 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:20.772894 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0067483d1a4a17832be0d902550788. Latest consensus state: current_term: 1 leader_uuid: "da0067483d1a4a17832be0d902550788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0067483d1a4a17832be0d902550788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40615 } } }
I20260806 16:16:20.773016 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:20.774986 7256 catalog_manager.cc:1347] Generated new cluster ID: 6ae3fdf5b5d940008176f19ff5c4da1b
I20260806 16:16:20.775043 7256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:20.786836 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:20.771410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34359 (local address 127.0.0.1:40615)
0806 16:16:20.771515 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:20.771519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:20.773201 (+ 1682us) server_negotiation.cc:407] Connection header received
0806 16:16:20.773689 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:20.773692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:20.773739 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:20.773813 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:20.774467 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:20.775233 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:20.781389 (+ 6156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:20.781780 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:20.781902 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:20.783462 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:20.783466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:20.783469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:20.783495 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:20.785163 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:20.785606 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:20.785610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:20.785612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:20.785696 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:20.785966 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:20.785969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:20.785971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:20.786191 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:20.786337 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:20.786546 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 16:16:20.786663 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:16:20.822696 7256 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:20.823733 7256 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:20.828521 7256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0067483d1a4a17832be0d902550788: Generated new TSK 0
I20260806 16:16:20.829382 7256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:20.836210 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7177
2026-08-06T16:16:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:20 dist-test-slave-0b58 krb5kdc[7278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:20 dist-test-slave-0b58 krb5kdc[7278](info): set up 2 sockets
Aug 06 16:16:20 dist-test-slave-0b58 krb5kdc[7278](info): commencing operation
krb5kdc: starting...
W20260806 16:16:21.864771 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:21 dist-test-slave-0b58 krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:21.992172 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:22.103120 7291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:22.103394 7291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:22.103466 7291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:22.107561 7291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:22.107625 7291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:22.107643 7291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:22.107699 7291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:22.107736 7291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:22.113107 7291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:22.114192 7291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:22.115216 7291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:22.124050 7297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:22.124125 7291 server_base.cc:1034] running on GCE node
W20260806 16:16:22.124120 7296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:22.124323 7299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:22.124897 7291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:22.126008 7291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:22.127202 7291 hybrid_clock.cc:648] HybridClock initialized: now 1786032982127186 us; error 27 us; skew 500 ppm
Aug 06 16:16:22 dist-test-slave-0b58 krb5kdc[7278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:22.129912 7291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:22.130787 7291 webserver.cc:460] Webserver started at http://127.0.0.1:33917/ using document root <none> and password file <none>
I20260806 16:16:22.131294 7291 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:22.131349 7291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:22.131506 7291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:22.133548 7291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "18d97f84e4e74d50a3848ad5ade3b87a"
format_stamp: "Formatted at 2026-08-06 16:16:22 on dist-test-slave-0b58"
server_key: "a2e1d343fc457a95ea7b88424fd3d48d"
server_key_iv: "4682d8150ece1044f788559d541fb166"
server_key_version: "encryptionkey@0"
I20260806 16:16:22.134064 7291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "18d97f84e4e74d50a3848ad5ade3b87a"
format_stamp: "Formatted at 2026-08-06 16:16:22 on dist-test-slave-0b58"
server_key: "a2e1d343fc457a95ea7b88424fd3d48d"
server_key_iv: "4682d8150ece1044f788559d541fb166"
server_key_version: "encryptionkey@0"
I20260806 16:16:22.137629 7291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:22.139859 7306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:22.141091 7291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:22.141237 7291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "18d97f84e4e74d50a3848ad5ade3b87a"
format_stamp: "Formatted at 2026-08-06 16:16:22 on dist-test-slave-0b58"
server_key: "a2e1d343fc457a95ea7b88424fd3d48d"
server_key_iv: "4682d8150ece1044f788559d541fb166"
server_key_version: "encryptionkey@0"
I20260806 16:16:22.141337 7291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:22.161916 7291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:22.164800 7291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:22.165009 7291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:22.172340 7291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43025
I20260806 16:16:22.172374 7361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43025 every 8 connection(s)
I20260806 16:16:22.173322 7291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:22.175992 7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:22.178638 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7291
I20260806 16:16:22.178786 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:22.179106 18792 external_mini_cluster.cc:1428] Setting key 88cbf969d66f50bfc051a26865f9fea7
I20260806 16:16:22.182243 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a: Bootstrap starting.
I20260806 16:16:22.184453 7362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:22.185199 7362 log.cc:824] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:22.187289 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a: No bootstrap required, opened a new log
I20260806 16:16:22.188057 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:22.180597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42415 (local address 127.0.0.1:43025)
0806 16:16:22.181030 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:22.181044 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:22.181079 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:16:22.181698 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:22.181715 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:22.182062 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:22.182401 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:22.182810 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:22.183764 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:22.184748 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:22.185237 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:22.185419 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:22.186404 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:22.186409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:22.186422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:22.186797 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:22.186851 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:22.186874 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:22.186988 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:22.187117 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 16:16:22.187329 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260806 16:16:22.190841 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:22.191018 7362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43025 } }
I20260806 16:16:22.191231 7362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:22.191304 7362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d97f84e4e74d50a3848ad5ade3b87a, State: Initialized, Role: FOLLOWER
I20260806 16:16:22.191735 7362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43025 } }
I20260806 16:16:22.191849 7362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:22.191929 7362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:22.192036 7362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:22.193142 7362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43025 } }
I20260806 16:16:22.193508 7362 leader_election.cc:302] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d97f84e4e74d50a3848ad5ade3b87a; no voters:
I20260806 16:16:22.193893 7362 leader_election.cc:288] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:22.194010 7372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:22.194213 7372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [term 1 LEADER]: Becoming Leader. State: Replica: 18d97f84e4e74d50a3848ad5ade3b87a, State: Running, Role: LEADER
I20260806 16:16:22.194502 7372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43025 } }
I20260806 16:16:22.195132 7362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:22.195997 7373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43025 } } }
I20260806 16:16:22.196136 7373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:22.196389 7373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d97f84e4e74d50a3848ad5ade3b87a. Latest consensus state: current_term: 1 leader_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d97f84e4e74d50a3848ad5ade3b87a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43025 } } }
I20260806 16:16:22.196478 7373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:22.197990 7381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:22.198123 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:22.193091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50257 (local address 127.0.0.1:43025)
0806 16:16:22.193216 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:22.193220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:22.193365 (+ 145us) server_negotiation.cc:407] Connection header received
0806 16:16:22.193478 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:22.193480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:22.193516 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:22.193581 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:22.193906 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:22.194638 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:22.195773 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:22.196310 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:22.196408 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:22.197457 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:22.197462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:22.197464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:22.197478 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:22.197507 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:22.197521 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:22.197588 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:22.197935 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 16:16:22.197997 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:16:22.200533 7381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:22.205878 7381 catalog_manager.cc:1347] Generated new cluster ID: 24b0065ab4784f6abdbe4b62d2a542d3
I20260806 16:16:22.205960 7381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:22.227882 7381 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:22.229362 7381 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:22.235457 7381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d97f84e4e74d50a3848ad5ade3b87a: Generated new TSK 0
I20260806 16:16:22.236147 7381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:22.241828 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7291
2026-08-06T16:16:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:22 dist-test-slave-0b58 krb5kdc[7394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:22 dist-test-slave-0b58 krb5kdc[7394](info): set up 2 sockets
Aug 06 16:16:22 dist-test-slave-0b58 krb5kdc[7394](info): commencing operation
krb5kdc: starting...
W20260806 16:16:23.269201 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:23 dist-test-slave-0b58 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:23.395805 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:23.506901 7407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:23.507194 7407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:23.507238 7407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:23.511281 7407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:23.511349 7407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:23.511368 7407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:23.511404 7407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:23.511420 7407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:23.517773 7407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:23.518869 7407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:23.519882 7407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:23.528623 7415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:23.528627 7412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:23.528707 7413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:23.528884 7407 server_base.cc:1034] running on GCE node
I20260806 16:16:23.529543 7407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:23.530643 7407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:23.531837 7407 hybrid_clock.cc:648] HybridClock initialized: now 1786032983531824 us; error 25 us; skew 500 ppm
Aug 06 16:16:23 dist-test-slave-0b58 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:23.534621 7407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:23.535540 7407 webserver.cc:460] Webserver started at http://127.0.0.1:42289/ using document root <none> and password file <none>
I20260806 16:16:23.536094 7407 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:23.536162 7407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:23.536419 7407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:23.538542 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "b3b6898137d54f429d11362995b63764"
format_stamp: "Formatted at 2026-08-06 16:16:23 on dist-test-slave-0b58"
server_key: "547b255869b029e85e57cb8ad5e9a323"
server_key_iv: "18e3110b6f92e6b07aec518cc9dd9d1c"
server_key_version: "encryptionkey@0"
I20260806 16:16:23.539067 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "b3b6898137d54f429d11362995b63764"
format_stamp: "Formatted at 2026-08-06 16:16:23 on dist-test-slave-0b58"
server_key: "547b255869b029e85e57cb8ad5e9a323"
server_key_iv: "18e3110b6f92e6b07aec518cc9dd9d1c"
server_key_version: "encryptionkey@0"
I20260806 16:16:23.542841 7407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 16:16:23.545264 7422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:23.546555 7407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:23.546701 7407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "b3b6898137d54f429d11362995b63764"
format_stamp: "Formatted at 2026-08-06 16:16:23 on dist-test-slave-0b58"
server_key: "547b255869b029e85e57cb8ad5e9a323"
server_key_iv: "18e3110b6f92e6b07aec518cc9dd9d1c"
server_key_version: "encryptionkey@0"
I20260806 16:16:23.546808 7407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:23.551661 7407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:23.554574 7407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:23.554774 7407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:23.562120 7407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39367
I20260806 16:16:23.562148 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39367 every 8 connection(s)
I20260806 16:16:23.563143 7407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:23.565933 7477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:23.571556 7477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764: Bootstrap starting.
I20260806 16:16:23.572850 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7407
I20260806 16:16:23.572990 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:23.573302 18792 external_mini_cluster.cc:1428] Setting key 7e510f72439a03c2747de1a0ffc38909
I20260806 16:16:23.574579 7477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:23.575392 7477 log.cc:824] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:23.577628 7477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764: No bootstrap required, opened a new log
I20260806 16:16:23.578658 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:23.574491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45137 (local address 127.0.0.1:39367)
0806 16:16:23.574981 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:23.574996 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:23.575034 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:16:23.575794 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:23.575822 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:23.576182 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:23.576615 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:23.577456 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:23.577461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:23.577472 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:23.577482 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:23.577536 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:23.577556 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:23.577676 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:23.577779 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 16:16:23.577940 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":152,"threads_started":1}
I20260806 16:16:23.581475 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:23.581646 7477 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b6898137d54f429d11362995b63764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260806 16:16:23.581908 7477 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:23.581995 7477 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b6898137d54f429d11362995b63764, State: Initialized, Role: FOLLOWER
I20260806 16:16:23.582429 7477 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b6898137d54f429d11362995b63764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260806 16:16:23.582579 7477 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:23.582666 7477 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:23.582787 7477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:23.583976 7477 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b6898137d54f429d11362995b63764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260806 16:16:23.584395 7477 leader_election.cc:302] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3b6898137d54f429d11362995b63764; no voters:
I20260806 16:16:23.584770 7477 leader_election.cc:288] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:23.585170 7487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:23.585441 7487 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [term 1 LEADER]: Becoming Leader. State: Replica: b3b6898137d54f429d11362995b63764, State: Running, Role: LEADER
I20260806 16:16:23.585695 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:23.583846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58533 (local address 127.0.0.1:39367)
0806 16:16:23.583964 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:23.583968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:23.584207 (+ 239us) server_negotiation.cc:407] Connection header received
0806 16:16:23.584371 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:23.584373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:23.584404 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:23.584469 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:23.585246 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:23.585250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:23.585253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:23.585264 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:23.585302 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:23.585316 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:23.585383 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:23.585546 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 16:16:23.585584 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:16:23.585855 7487 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b6898137d54f429d11362995b63764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260806 16:16:23.586236 7477 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:23.588204 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b6898137d54f429d11362995b63764. Latest consensus state: current_term: 1 leader_uuid: "b3b6898137d54f429d11362995b63764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b6898137d54f429d11362995b63764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260806 16:16:23.588233 7488 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3b6898137d54f429d11362995b63764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b6898137d54f429d11362995b63764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260806 16:16:23.588389 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:23.588392 7488 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764 [sys.catalog]: This master's current role is: LEADER
W20260806 16:16:23.591053 7501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:23.591141 7501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:16:23.591204 7502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:23.591986 7502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:23.597533 7502 catalog_manager.cc:1347] Generated new cluster ID: 237135a8fcaa40d6b65a8ada2649f496
I20260806 16:16:23.597604 7502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:23.610419 7502 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:23.610769 7502 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:23.620919 7502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3b6898137d54f429d11362995b63764: Generated new TSK 0
I20260806 16:16:23.621752 7502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:23.631263 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7407
2026-08-06T16:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:23 dist-test-slave-0b58 krb5kdc[7508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:23 dist-test-slave-0b58 krb5kdc[7508](info): set up 2 sockets
Aug 06 16:16:23 dist-test-slave-0b58 krb5kdc[7508](info): commencing operation
krb5kdc: starting...
W20260806 16:16:24.668766 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:24 dist-test-slave-0b58 krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:24.795279 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:24.911243 7521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:24.911527 7521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:24.911571 7521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:24.915611 7521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:24.915685 7521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:24.915705 7521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:24.915722 7521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:24.915738 7521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:24.921188 7521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:24.922235 7521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:24.923237 7521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:24.931676 7530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:24.931689 7526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:24.931767 7528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:24.932037 7521 server_base.cc:1034] running on GCE node
I20260806 16:16:24.932646 7521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:24.933763 7521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:24.934924 7521 hybrid_clock.cc:648] HybridClock initialized: now 1786032984934894 us; error 46 us; skew 500 ppm
Aug 06 16:16:24 dist-test-slave-0b58 krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:24.937824 7521 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:24.938652 7521 webserver.cc:460] Webserver started at http://127.0.0.1:34807/ using document root <none> and password file <none>
I20260806 16:16:24.939167 7521 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:24.939224 7521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:24.939393 7521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:24.941452 7521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "41d5e827401e467bbcfc25f80351f063"
format_stamp: "Formatted at 2026-08-06 16:16:24 on dist-test-slave-0b58"
server_key: "21b17e334716985e5380a2114de10a34"
server_key_iv: "a08f015de277a9890c8a4bbea361c870"
server_key_version: "encryptionkey@0"
I20260806 16:16:24.941982 7521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "41d5e827401e467bbcfc25f80351f063"
format_stamp: "Formatted at 2026-08-06 16:16:24 on dist-test-slave-0b58"
server_key: "21b17e334716985e5380a2114de10a34"
server_key_iv: "a08f015de277a9890c8a4bbea361c870"
server_key_version: "encryptionkey@0"
I20260806 16:16:24.945712 7521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:16:24.948088 7537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:24.949414 7521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:24.949545 7521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "41d5e827401e467bbcfc25f80351f063"
format_stamp: "Formatted at 2026-08-06 16:16:24 on dist-test-slave-0b58"
server_key: "21b17e334716985e5380a2114de10a34"
server_key_iv: "a08f015de277a9890c8a4bbea361c870"
server_key_version: "encryptionkey@0"
I20260806 16:16:24.949656 7521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:24.960973 7521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:24.964092 7521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:24.964300 7521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:24.971007 7521 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37927
I20260806 16:16:24.971098 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37927 every 8 connection(s)
I20260806 16:16:24.972083 7521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:24.975445 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:24.982357 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7521
I20260806 16:16:24.982510 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:24.982805 18792 external_mini_cluster.cc:1428] Setting key 0b9b54196d3cb27479aa883b67cb201e
I20260806 16:16:24.983505 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063: Bootstrap starting.
I20260806 16:16:24.986155 7590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:24.987044 7590 log.cc:824] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:24.989274 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063: No bootstrap required, opened a new log
Aug 06 16:16:24 dist-test-slave-0b58 krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:16:24.993731 7590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d5e827401e467bbcfc25f80351f063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260806 16:16:24.994038 7590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:24.994109 7590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d5e827401e467bbcfc25f80351f063, State: Initialized, Role: FOLLOWER
I20260806 16:16:24.994630 7590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d5e827401e467bbcfc25f80351f063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260806 16:16:24.994782 7590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:24.994860 7590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:24.995000 7590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:24.995927 7590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d5e827401e467bbcfc25f80351f063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260806 16:16:24.996316 7590 leader_election.cc:302] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d5e827401e467bbcfc25f80351f063; no voters:
I20260806 16:16:24.996703 7590 leader_election.cc:288] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:24.996815 7595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:24.997073 7595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [term 1 LEADER]: Becoming Leader. State: Replica: 41d5e827401e467bbcfc25f80351f063, State: Running, Role: LEADER
I20260806 16:16:24.997455 7595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d5e827401e467bbcfc25f80351f063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260806 16:16:24.998378 7590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:24.998721 7595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d5e827401e467bbcfc25f80351f063. Latest consensus state: current_term: 1 leader_uuid: "41d5e827401e467bbcfc25f80351f063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d5e827401e467bbcfc25f80351f063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } } }
I20260806 16:16:24.998888 7595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:24.999157 7596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d5e827401e467bbcfc25f80351f063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d5e827401e467bbcfc25f80351f063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } } }
I20260806 16:16:24.999274 7596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:24.999301 7599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:24.999330 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:24.984770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:53077 (local address 127.0.0.1:37927)
0806 16:16:24.985242 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:24.985266 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:24.985303 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:16:24.986002 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:24.986022 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:24.986439 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:24.986726 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:24.987704 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:24.988849 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:24.990007 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:24.990471 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:24.990596 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:24.993524 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:24.993529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:24.993543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:24.993571 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:24.995601 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:24.996182 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:24.996187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:24.996192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:24.996370 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:24.996806 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:24.996811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:24.996813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:24.997387 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:24.997591 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:24.998094 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 16:16:24.998325 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260806 16:16:25.002530 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:25.004225 7599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:25.010869 7599 catalog_manager.cc:1347] Generated new cluster ID: 3124a6468c1441eca10329fb83e19e39
I20260806 16:16:25.010939 7599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:25.018961 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:25.008956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:51655 (local address 127.0.0.1:37927)
0806 16:16:25.009068 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:25.009073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:25.009750 (+ 677us) server_negotiation.cc:407] Connection header received
0806 16:16:25.009881 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:25.009884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:25.009937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:25.010026 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:25.010952 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:25.011796 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:25.013099 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:25.013471 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:25.013563 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:25.015343 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:25.015346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:25.015349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:25.015374 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:25.017097 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:25.017652 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:25.017655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:25.017657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:25.017736 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:25.018080 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:25.018083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:25.018085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:25.018274 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:25.018397 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:25.018688 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 16:16:25.018799 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:16:25.027175 7599 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:25.028475 7599 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:25.035838 7599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d5e827401e467bbcfc25f80351f063: Generated new TSK 0
I20260806 16:16:25.036674 7599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:25.045203 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7521
W20260806 16:16:25.051303 7506 connection.cc:535] client connection to 127.0.0.1:37927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37927): Connection reset by peer (error 104)
2026-08-06T16:16:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:25 dist-test-slave-0b58 krb5kdc[7621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:25 dist-test-slave-0b58 krb5kdc[7621](info): set up 2 sockets
Aug 06 16:16:25 dist-test-slave-0b58 krb5kdc[7621](info): commencing operation
krb5kdc: starting...
W20260806 16:16:26.073083 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:26 dist-test-slave-0b58 krb5kdc[7621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:26.198547 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:26.312441 7634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:26.312707 7634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:26.312804 7634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:26.316792 7634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:26.316849 7634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:26.316870 7634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:26.316919 7634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:26.316952 7634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:26.322186 7634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:26.323279 7634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:26.324393 7634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:26.332899 7642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:26.332901 7639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:26.332899 7640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:26.333194 7634 server_base.cc:1034] running on GCE node
I20260806 16:16:26.333843 7634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:26.334916 7634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:26.336081 7634 hybrid_clock.cc:648] HybridClock initialized: now 1786032986336061 us; error 28 us; skew 500 ppm
Aug 06 16:16:26 dist-test-slave-0b58 krb5kdc[7621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:26.344099 7634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:26.345111 7634 webserver.cc:460] Webserver started at http://127.0.0.1:32983/ using document root <none> and password file <none>
I20260806 16:16:26.345691 7634 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:26.345794 7634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:26.346026 7634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:26.348073 7634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e"
format_stamp: "Formatted at 2026-08-06 16:16:26 on dist-test-slave-0b58"
server_key: "3b380277184833c5cff5b1cbe0b7294a"
server_key_iv: "fb022a7ec29cfcf30b86c18d44717b00"
server_key_version: "encryptionkey@0"
I20260806 16:16:26.348714 7634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e"
format_stamp: "Formatted at 2026-08-06 16:16:26 on dist-test-slave-0b58"
server_key: "3b380277184833c5cff5b1cbe0b7294a"
server_key_iv: "fb022a7ec29cfcf30b86c18d44717b00"
server_key_version: "encryptionkey@0"
I20260806 16:16:26.352494 7634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:26.354868 7649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:26.356172 7634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:26.356345 7634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e"
format_stamp: "Formatted at 2026-08-06 16:16:26 on dist-test-slave-0b58"
server_key: "3b380277184833c5cff5b1cbe0b7294a"
server_key_iv: "fb022a7ec29cfcf30b86c18d44717b00"
server_key_version: "encryptionkey@0"
I20260806 16:16:26.356472 7634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:26.366348 7634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:26.369674 7634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:26.369889 7634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:26.377032 7634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44951
I20260806 16:16:26.377054 7702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44951 every 8 connection(s)
I20260806 16:16:26.377997 7634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:26.380771 7704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:26.385226 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7634
I20260806 16:16:26.385363 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:26.385637 18792 external_mini_cluster.cc:1428] Setting key 1112285d326219efe5df9be1ca9d0360
I20260806 16:16:26.386528 7704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e: Bootstrap starting.
I20260806 16:16:26.388834 7704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:26.389669 7704 log.cc:824] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:26.391811 7704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e: No bootstrap required, opened a new log
I20260806 16:16:26.394850 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:26.387105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:58585 (local address 127.0.0.1:44951)
0806 16:16:26.387525 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:26.387542 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:26.387577 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:16:26.388193 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:26.388210 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:26.388624 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:26.388934 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:26.389304 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:26.390336 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:26.391520 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:26.392050 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:26.392167 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:26.393106 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:26.393112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:26.393124 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:26.393508 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:26.393559 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:26.393580 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:26.393680 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:26.393914 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 16:16:26.394075 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":143,"threads_started":1}
I20260806 16:16:26.395910 7704 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44951 } }
I20260806 16:16:26.396188 7704 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:26.396277 7704 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2f45c23d1c4c41927f77c9ff87fd4e, State: Initialized, Role: FOLLOWER
I20260806 16:16:26.396737 7704 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44951 } }
I20260806 16:16:26.396917 7704 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:26.397001 7704 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:26.397099 7704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:26.398005 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:26.398308 7704 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44951 } }
I20260806 16:16:26.398687 7704 leader_election.cc:302] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2f45c23d1c4c41927f77c9ff87fd4e; no voters:
I20260806 16:16:26.399026 7704 leader_election.cc:288] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:26.399132 7709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:26.399386 7709 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [term 1 LEADER]: Becoming Leader. State: Replica: dc2f45c23d1c4c41927f77c9ff87fd4e, State: Running, Role: LEADER
I20260806 16:16:26.399739 7709 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44951 } }
I20260806 16:16:26.400449 7704 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:26.400990 7709 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2f45c23d1c4c41927f77c9ff87fd4e. Latest consensus state: current_term: 1 leader_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44951 } } }
I20260806 16:16:26.401163 7709 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:26.401500 7710 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2f45c23d1c4c41927f77c9ff87fd4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44951 } } }
I20260806 16:16:26.401602 7710 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:26.401634 7720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:26.404799 7720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:26.407517 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:26.400370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:38485 (local address 127.0.0.1:44951)
0806 16:16:26.400481 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:26.400485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:26.402831 (+ 2346us) server_negotiation.cc:407] Connection header received
0806 16:16:26.402995 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:26.402999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:26.403046 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:26.403126 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:26.403444 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:26.404198 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:26.405595 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:26.405931 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:26.406039 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:26.407015 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:26.407019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:26.407022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:26.407035 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:26.407063 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:26.407076 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:26.407168 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:26.407354 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 16:16:26.407389 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:26.410763 7720 catalog_manager.cc:1347] Generated new cluster ID: 4da6a4dad38941daa930f6f0bbca6360
I20260806 16:16:26.410826 7720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:26.420941 7720 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:26.421845 7720 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:26.428200 7720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc2f45c23d1c4c41927f77c9ff87fd4e: Generated new TSK 0
I20260806 16:16:26.428893 7720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:26.437688 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7634
W20260806 16:16:26.444563 7619 connection.cc:535] client connection to 127.0.0.1:44951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44951): Connection reset by peer (error 104)
2026-08-06T16:16:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:26 dist-test-slave-0b58 krb5kdc[7737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:26 dist-test-slave-0b58 krb5kdc[7737](info): set up 2 sockets
Aug 06 16:16:26 dist-test-slave-0b58 krb5kdc[7737](info): commencing operation
krb5kdc: starting...
W20260806 16:16:27.472903 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:27 dist-test-slave-0b58 krb5kdc[7737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:27.599825 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:27.713265 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:27.713548 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:27.713591 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:27.717674 7751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:27.717736 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:27.717757 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:27.717774 7751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:27.717789 7751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:27.723070 7751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:27.724134 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:27.725143 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:27.733718 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:27.733754 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:27.733762 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:27.734040 7751 server_base.cc:1034] running on GCE node
I20260806 16:16:27.734591 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:27.735672 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:27.736856 7751 hybrid_clock.cc:648] HybridClock initialized: now 1786032987736848 us; error 31 us; skew 500 ppm
Aug 06 16:16:27 dist-test-slave-0b58 krb5kdc[7737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:27.739475 7751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:27.740407 7751 webserver.cc:460] Webserver started at http://127.0.0.1:34597/ using document root <none> and password file <none>
I20260806 16:16:27.740962 7751 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:27.741055 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:27.741297 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:27.743358 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "678380fc9a85495788625181f2e1b781"
format_stamp: "Formatted at 2026-08-06 16:16:27 on dist-test-slave-0b58"
server_key: "85ed7b8b81d80729063960d468f74296"
server_key_iv: "59fb5dc7461cbfee9079702ac56099cc"
server_key_version: "encryptionkey@0"
I20260806 16:16:27.743935 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "678380fc9a85495788625181f2e1b781"
format_stamp: "Formatted at 2026-08-06 16:16:27 on dist-test-slave-0b58"
server_key: "85ed7b8b81d80729063960d468f74296"
server_key_iv: "59fb5dc7461cbfee9079702ac56099cc"
server_key_version: "encryptionkey@0"
I20260806 16:16:27.747630 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 16:16:27.750046 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:27.751320 7751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:27.751461 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "678380fc9a85495788625181f2e1b781"
format_stamp: "Formatted at 2026-08-06 16:16:27 on dist-test-slave-0b58"
server_key: "85ed7b8b81d80729063960d468f74296"
server_key_iv: "59fb5dc7461cbfee9079702ac56099cc"
server_key_version: "encryptionkey@0"
I20260806 16:16:27.751577 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:27.756165 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:27.759136 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:27.759337 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:27.766532 7751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36045
I20260806 16:16:27.766570 7821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36045 every 8 connection(s)
I20260806 16:16:27.767477 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:27.770243 7822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:27.775641 7822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781: Bootstrap starting.
I20260806 16:16:27.776623 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7751
I20260806 16:16:27.776770 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:27.777112 18792 external_mini_cluster.cc:1428] Setting key afc751a1abf22d032c134afe42dd68bc
I20260806 16:16:27.778664 7822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:27.779400 7822 log.cc:824] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:27.781507 7822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781: No bootstrap required, opened a new log
I20260806 16:16:27.782193 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:27.778227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:44553 (local address 127.0.0.1:36045)
0806 16:16:27.778690 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:27.778705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:27.778739 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:16:27.779417 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:27.779435 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:27.779774 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:27.780101 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:27.780952 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:27.780956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:27.780968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:27.780980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:27.781032 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:27.781051 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:27.781170 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:27.781297 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 16:16:27.781460 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":133,"threads_started":1}
I20260806 16:16:27.785027 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:27.785681 7822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678380fc9a85495788625181f2e1b781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260806 16:16:27.785920 7822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:27.785998 7822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678380fc9a85495788625181f2e1b781, State: Initialized, Role: FOLLOWER
I20260806 16:16:27.786523 7822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678380fc9a85495788625181f2e1b781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260806 16:16:27.786689 7822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:27.786757 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:27.786855 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:27.788074 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678380fc9a85495788625181f2e1b781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260806 16:16:27.788486 7822 leader_election.cc:302] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 678380fc9a85495788625181f2e1b781; no voters:
I20260806 16:16:27.788846 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:27.788930 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:27.788902 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:27.787169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:36589 (local address 127.0.0.1:36045)
0806 16:16:27.787276 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:27.787280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:27.787505 (+ 225us) server_negotiation.cc:407] Connection header received
0806 16:16:27.787617 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:27.787619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:27.787645 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:27.787710 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:27.788501 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:27.788504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:27.788506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:27.788516 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:27.788539 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:27.788549 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:27.788645 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:27.788734 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 16:16:27.788766 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:16:27.789417 7832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [term 1 LEADER]: Becoming Leader. State: Replica: 678380fc9a85495788625181f2e1b781, State: Running, Role: LEADER
I20260806 16:16:27.789902 7832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678380fc9a85495788625181f2e1b781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260806 16:16:27.790439 7822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:27.791673 7833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 678380fc9a85495788625181f2e1b781. Latest consensus state: current_term: 1 leader_uuid: "678380fc9a85495788625181f2e1b781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678380fc9a85495788625181f2e1b781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } } }
I20260806 16:16:27.791707 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "678380fc9a85495788625181f2e1b781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678380fc9a85495788625181f2e1b781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } } }
I20260806 16:16:27.791846 7833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:27.791867 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:27.793000 7842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:27.796032 7842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:27.801060 7842 catalog_manager.cc:1347] Generated new cluster ID: 4f039b92e9f648af872e7a39090d89d7
I20260806 16:16:27.801128 7842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:27.812820 7842 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:27.813159 7842 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:27.816864 7842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 678380fc9a85495788625181f2e1b781: Generated new TSK 0
I20260806 16:16:27.817615 7842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:27.820410 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7751
2026-08-06T16:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:27 dist-test-slave-0b58 krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:27 dist-test-slave-0b58 krb5kdc[7854](info): set up 2 sockets
Aug 06 16:16:27 dist-test-slave-0b58 krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260806 16:16:28.860569 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:28 dist-test-slave-0b58 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:28.986982 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:29.096514 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:29.096755 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:29.096814 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:29.100699 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:29.100754 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:29.100771 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:29.100787 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:29.100801 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:29.105924 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:29.106925 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:29.107887 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:29.116506 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:29.116560 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:29.116551 7867 server_base.cc:1034] running on GCE node
W20260806 16:16:29.116484 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:29.117314 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:29.118423 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:29.119618 7867 hybrid_clock.cc:648] HybridClock initialized: now 1786032989119600 us; error 30 us; skew 500 ppm
Aug 06 16:16:29 dist-test-slave-0b58 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:29.122437 7867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:29.123310 7867 webserver.cc:460] Webserver started at http://127.0.0.1:44949/ using document root <none> and password file <none>
I20260806 16:16:29.123905 7867 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:29.123999 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:29.124228 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:29.126323 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "11849e3ceca54a629380a017b9aec08b"
format_stamp: "Formatted at 2026-08-06 16:16:29 on dist-test-slave-0b58"
server_key: "e976b4d80f6cb5eb240a7e01c71344d8"
server_key_iv: "c801017330cea590369671831d1745dc"
server_key_version: "encryptionkey@0"
I20260806 16:16:29.126912 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "11849e3ceca54a629380a017b9aec08b"
format_stamp: "Formatted at 2026-08-06 16:16:29 on dist-test-slave-0b58"
server_key: "e976b4d80f6cb5eb240a7e01c71344d8"
server_key_iv: "c801017330cea590369671831d1745dc"
server_key_version: "encryptionkey@0"
I20260806 16:16:29.130686 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 16:16:29.133112 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:29.134699 7867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 16:16:29.134845 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "11849e3ceca54a629380a017b9aec08b"
format_stamp: "Formatted at 2026-08-06 16:16:29 on dist-test-slave-0b58"
server_key: "e976b4d80f6cb5eb240a7e01c71344d8"
server_key_iv: "c801017330cea590369671831d1745dc"
server_key_version: "encryptionkey@0"
I20260806 16:16:29.134961 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:29.157207 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:29.160146 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:29.160395 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:29.167719 7867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34479
I20260806 16:16:29.167742 7936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34479 every 8 connection(s)
I20260806 16:16:29.168712 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:29.171489 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:29.173326 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7867
I20260806 16:16:29.173478 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:29.173787 18792 external_mini_cluster.cc:1428] Setting key c35c9ef225469fc10e20542bed396ef2
I20260806 16:16:29.178517 7937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b: Bootstrap starting.
I20260806 16:16:29.181190 7937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:29.181989 7937 log.cc:824] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:16:29 dist-test-slave-0b58 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:16:29.184077 7937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b: No bootstrap required, opened a new log
I20260806 16:16:29.188902 7937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11849e3ceca54a629380a017b9aec08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34479 } }
I20260806 16:16:29.189178 7937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:29.189237 7937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11849e3ceca54a629380a017b9aec08b, State: Initialized, Role: FOLLOWER
I20260806 16:16:29.189365 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:29.175238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:47483 (local address 127.0.0.1:34479)
0806 16:16:29.175666 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:29.175686 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:29.175730 (+ 44us) server_negotiation.cc:407] Connection header received
0806 16:16:29.176599 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:29.176627 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:29.177070 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:29.177431 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:29.178107 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:29.179536 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:29.180455 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:29.181142 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:29.181311 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:29.184109 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:29.184118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:29.184140 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:29.184170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:29.186155 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:29.186677 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:29.186684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:29.186689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:29.186831 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:29.187147 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:29.187150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:29.187152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:29.187660 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:29.187879 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:29.188104 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 16:16:29.188359 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260806 16:16:29.189790 7937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11849e3ceca54a629380a017b9aec08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34479 } }
I20260806 16:16:29.189955 7937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:29.190047 7937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:29.190178 7937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:29.191073 7937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11849e3ceca54a629380a017b9aec08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34479 } }
I20260806 16:16:29.191421 7937 leader_election.cc:302] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11849e3ceca54a629380a017b9aec08b; no voters:
I20260806 16:16:29.191757 7937 leader_election.cc:288] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:29.191913 7942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:29.192565 7942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [term 1 LEADER]: Becoming Leader. State: Replica: 11849e3ceca54a629380a017b9aec08b, State: Running, Role: LEADER
I20260806 16:16:29.192875 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:29.192960 7942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11849e3ceca54a629380a017b9aec08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34479 } }
I20260806 16:16:29.193131 7937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:29.194131 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11849e3ceca54a629380a017b9aec08b. Latest consensus state: current_term: 1 leader_uuid: "11849e3ceca54a629380a017b9aec08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11849e3ceca54a629380a017b9aec08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34479 } } }
I20260806 16:16:29.194276 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:29.194664 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:29.195148 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11849e3ceca54a629380a017b9aec08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11849e3ceca54a629380a017b9aec08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34479 } } }
I20260806 16:16:29.195246 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:29.198163 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:29.204095 7955 catalog_manager.cc:1347] Generated new cluster ID: f807b0a662d94a4aadb2cb81843356c0
I20260806 16:16:29.204166 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:29.205224 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:29.195472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:45005 (local address 127.0.0.1:34479)
0806 16:16:29.195577 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:29.195583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:29.195916 (+ 333us) server_negotiation.cc:407] Connection header received
0806 16:16:29.196047 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:29.196050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:29.196122 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:29.196220 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:29.197025 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:29.197775 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:29.198903 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:29.199237 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:29.199360 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:29.201346 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:29.201350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:29.201352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:29.201376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:29.202958 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:29.203450 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:29.203453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:29.203455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:29.203538 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:29.203879 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:29.203883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:29.203884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:29.204110 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:29.204887 (+ 777us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:29.204988 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 16:16:29.205081 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 16:16:29.211790 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:29.212868 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:29.218798 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11849e3ceca54a629380a017b9aec08b: Generated new TSK 0
I20260806 16:16:29.219442 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:29.224238 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7867
W20260806 16:16:29.230654 7852 connection.cc:535] client connection to 127.0.0.1:34479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34479): Connection reset by peer (error 104)
2026-08-06T16:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:29 dist-test-slave-0b58 krb5kdc[7969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:29 dist-test-slave-0b58 krb5kdc[7969](info): set up 2 sockets
Aug 06 16:16:29 dist-test-slave-0b58 krb5kdc[7969](info): commencing operation
krb5kdc: starting...
W20260806 16:16:30.269160 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:30 dist-test-slave-0b58 krb5kdc[7969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:30.397022 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:30.522204 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:30.522521 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:30.522648 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:30.526793 7982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:30.526889 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:30.526932 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:30.526989 7982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:30.527029 7982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:30.532404 7982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:30.533509 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:30.534564 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:30.543190 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:30.543365 7982 server_base.cc:1034] running on GCE node
W20260806 16:16:30.543190 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:30.543553 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:30.544073 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:30.545173 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:30.546340 7982 hybrid_clock.cc:648] HybridClock initialized: now 1786032990546321 us; error 29 us; skew 500 ppm
Aug 06 16:16:30 dist-test-slave-0b58 krb5kdc[7969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:30.549069 7982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:30.549983 7982 webserver.cc:460] Webserver started at http://127.0.0.1:41335/ using document root <none> and password file <none>
I20260806 16:16:30.550534 7982 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:30.550593 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:30.550819 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:30.552891 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "c0e8c5a48c344191952a2ba9dc93224e"
format_stamp: "Formatted at 2026-08-06 16:16:30 on dist-test-slave-0b58"
server_key: "78c739c591992a647a1fa4da1ff5114e"
server_key_iv: "dd86e5f8117b1589d919ca7915e9d974"
server_key_version: "encryptionkey@0"
I20260806 16:16:30.553452 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "c0e8c5a48c344191952a2ba9dc93224e"
format_stamp: "Formatted at 2026-08-06 16:16:30 on dist-test-slave-0b58"
server_key: "78c739c591992a647a1fa4da1ff5114e"
server_key_iv: "dd86e5f8117b1589d919ca7915e9d974"
server_key_version: "encryptionkey@0"
I20260806 16:16:30.557217 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:30.559618 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:30.561076 7982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:30.561252 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "c0e8c5a48c344191952a2ba9dc93224e"
format_stamp: "Formatted at 2026-08-06 16:16:30 on dist-test-slave-0b58"
server_key: "78c739c591992a647a1fa4da1ff5114e"
server_key_iv: "dd86e5f8117b1589d919ca7915e9d974"
server_key_version: "encryptionkey@0"
I20260806 16:16:30.561367 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:30.573225 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:30.576196 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:30.576442 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:30.583305 7982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36509
I20260806 16:16:30.583333 8051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36509 every 8 connection(s)
I20260806 16:16:30.584249 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:30.587103 8052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:30.592649 8052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e: Bootstrap starting.
I20260806 16:16:30.594173 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 7982
I20260806 16:16:30.594316 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:30.594604 18792 external_mini_cluster.cc:1428] Setting key 52ed13efbbb3004e50358ef035df3b64
I20260806 16:16:30.595372 8052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:30.596293 8052 log.cc:824] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:30.598572 8052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e: No bootstrap required, opened a new log
I20260806 16:16:30.602846 8052 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e8c5a48c344191952a2ba9dc93224e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260806 16:16:30.603117 8052 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:30.603200 8052 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e8c5a48c344191952a2ba9dc93224e, State: Initialized, Role: FOLLOWER
I20260806 16:16:30.603627 8052 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e8c5a48c344191952a2ba9dc93224e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260806 16:16:30.603780 8052 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:30.603870 8052 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:30.603967 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:30.595747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:52671 (local address 127.0.0.1:36509)
0806 16:16:30.596170 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:30.596186 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:30.596224 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:16:30.597124 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:30.597145 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:30.597474 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:30.597752 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:30.598149 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:30.599177 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:30.600423 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:30.600895 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:30.601018 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:30.602019 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:30.602024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:30.602037 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:30.602420 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:30.602475 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:30.602497 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:30.602597 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:30.602823 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 16:16:30.602994 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
I20260806 16:16:30.603968 8052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:30.605217 8052 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e8c5a48c344191952a2ba9dc93224e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260806 16:16:30.605564 8052 leader_election.cc:302] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0e8c5a48c344191952a2ba9dc93224e; no voters:
I20260806 16:16:30.605916 8052 leader_election.cc:288] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:30.606043 8057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:30.606276 8057 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [term 1 LEADER]: Becoming Leader. State: Replica: c0e8c5a48c344191952a2ba9dc93224e, State: Running, Role: LEADER
I20260806 16:16:30.606600 8057 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e8c5a48c344191952a2ba9dc93224e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260806 16:16:30.607136 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:30.607295 8052 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:30.610276 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0e8c5a48c344191952a2ba9dc93224e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e8c5a48c344191952a2ba9dc93224e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } } }
I20260806 16:16:30.610287 8059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0e8c5a48c344191952a2ba9dc93224e. Latest consensus state: current_term: 1 leader_uuid: "c0e8c5a48c344191952a2ba9dc93224e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e8c5a48c344191952a2ba9dc93224e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } } }
I20260806 16:16:30.610440 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:30.610450 8059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:30.610832 8078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:30.613722 8078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:30.614404 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:30.609464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:35775 (local address 127.0.0.1:36509)
0806 16:16:30.609723 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:30.609728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:30.609809 (+ 81us) server_negotiation.cc:407] Connection header received
0806 16:16:30.610060 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:30.610065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:30.610121 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:30.610209 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:30.610545 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:30.611366 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:30.612340 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:30.612675 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:30.612763 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:30.613872 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:30.613876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:30.613878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:30.613893 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:30.613922 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:30.613935 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:30.613993 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:30.614230 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 16:16:30.614271 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260806 16:16:30.619498 8078 catalog_manager.cc:1347] Generated new cluster ID: 5f992bd43ae842b49512c43d9d918011
I20260806 16:16:30.619568 8078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:30.625234 8078 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:30.626444 8078 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:30.639839 8078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0e8c5a48c344191952a2ba9dc93224e: Generated new TSK 0
I20260806 16:16:30.640630 8078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:30.649432 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 7982
W20260806 16:16:30.655925 7967 connection.cc:535] client connection to 127.0.0.1:36509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36509): Connection reset by peer (error 104)
2026-08-06T16:16:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:30 dist-test-slave-0b58 krb5kdc[8084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:30 dist-test-slave-0b58 krb5kdc[8084](info): set up 2 sockets
Aug 06 16:16:30 dist-test-slave-0b58 krb5kdc[8084](info): commencing operation
krb5kdc: starting...
W20260806 16:16:31.696786 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:31 dist-test-slave-0b58 krb5kdc[8084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:16:31.823813 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:31.939049 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:31.939381 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:31.939476 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:31.943562 8098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:31.943667 8098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:31.943707 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:31.943778 8098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:31.943840 8098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:31.949565 8098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:31.950830 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:31.951951 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:31.960731 8106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:31.960737 8103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:31.960737 8104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:31.960916 8098 server_base.cc:1034] running on GCE node
I20260806 16:16:31.961706 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:31.962821 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:31.963980 8098 hybrid_clock.cc:648] HybridClock initialized: now 1786032991963968 us; error 24 us; skew 500 ppm
Aug 06 16:16:31 dist-test-slave-0b58 krb5kdc[8084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:31.972368 8098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:16:31.973346 8098 webserver.cc:460] Webserver started at http://127.0.0.1:39357/ using document root <none> and password file <none>
I20260806 16:16:31.973946 8098 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:31.974011 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:31.974254 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:31.976495 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "84f2ebd9416544cc98927546c2bd7a87"
format_stamp: "Formatted at 2026-08-06 16:16:31 on dist-test-slave-0b58"
server_key: "8256bc4a0d74a1da0acab6ee564aee28"
server_key_iv: "3e8312a1de29e167ff58dc0a41ae044e"
server_key_version: "encryptionkey@0"
I20260806 16:16:31.977116 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "84f2ebd9416544cc98927546c2bd7a87"
format_stamp: "Formatted at 2026-08-06 16:16:31 on dist-test-slave-0b58"
server_key: "8256bc4a0d74a1da0acab6ee564aee28"
server_key_iv: "3e8312a1de29e167ff58dc0a41ae044e"
server_key_version: "encryptionkey@0"
I20260806 16:16:31.981019 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:16:31.983503 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:31.984861 8098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:31.984965 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "84f2ebd9416544cc98927546c2bd7a87"
format_stamp: "Formatted at 2026-08-06 16:16:31 on dist-test-slave-0b58"
server_key: "8256bc4a0d74a1da0acab6ee564aee28"
server_key_iv: "3e8312a1de29e167ff58dc0a41ae044e"
server_key_version: "encryptionkey@0"
I20260806 16:16:31.985055 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:31.989655 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:31.992659 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:31.992808 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:31.999754 8098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44197
I20260806 16:16:31.999778 8166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44197 every 8 connection(s)
I20260806 16:16:32.000900 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:32.003769 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:32.009564 8167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87: Bootstrap starting.
I20260806 16:16:32.010983 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8098
I20260806 16:16:32.011138 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:32.011412 18792 external_mini_cluster.cc:1428] Setting key a87c9660275e8bf020e09cc47c60c402
I20260806 16:16:32.012195 8167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:32.013127 8167 log.cc:824] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:32.015195 8167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87: No bootstrap required, opened a new log
I20260806 16:16:32.016884 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:32.012591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:49849 (local address 127.0.0.1:44197)
0806 16:16:32.013059 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:32.013074 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:32.013111 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:16:32.014054 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:32.014076 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:32.014374 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:32.014718 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:32.015596 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:32.015601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:32.015614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:32.015624 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:32.015680 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:32.015699 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:32.015825 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:32.015950 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 16:16:32.016118 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":144,"threads_started":1}
I20260806 16:16:32.019332 8167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f2ebd9416544cc98927546c2bd7a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44197 } }
I20260806 16:16:32.019620 8167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:32.019678 8167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f2ebd9416544cc98927546c2bd7a87, State: Initialized, Role: FOLLOWER
I20260806 16:16:32.020020 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:32.020195 8167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f2ebd9416544cc98927546c2bd7a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44197 } }
I20260806 16:16:32.020404 8167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:32.020474 8167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:32.020572 8167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:32.021625 8167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f2ebd9416544cc98927546c2bd7a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44197 } }
I20260806 16:16:32.021989 8167 leader_election.cc:302] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f2ebd9416544cc98927546c2bd7a87; no voters:
I20260806 16:16:32.022356 8167 leader_election.cc:288] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:32.022490 8176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:32.022708 8176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [term 1 LEADER]: Becoming Leader. State: Replica: 84f2ebd9416544cc98927546c2bd7a87, State: Running, Role: LEADER
I20260806 16:16:32.023001 8176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f2ebd9416544cc98927546c2bd7a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44197 } }
I20260806 16:16:32.023847 8167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:32.024322 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:32.022257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:42237 (local address 127.0.0.1:44197)
0806 16:16:32.022369 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:32.022374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:32.022578 (+ 204us) server_negotiation.cc:407] Connection header received
0806 16:16:32.022734 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:32.022738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:32.022775 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:32.022844 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:32.023806 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:32.023809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:32.023812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:32.023824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:32.023854 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:32.023867 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:32.023953 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:32.024133 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 16:16:32.024177 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:16:32.024295 8176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f2ebd9416544cc98927546c2bd7a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f2ebd9416544cc98927546c2bd7a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44197 } } }
I20260806 16:16:32.024403 8176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:32.024619 8176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f2ebd9416544cc98927546c2bd7a87. Latest consensus state: current_term: 1 leader_uuid: "84f2ebd9416544cc98927546c2bd7a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f2ebd9416544cc98927546c2bd7a87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44197 } } }
I20260806 16:16:32.024706 8176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87 [sys.catalog]: This master's current role is: LEADER
W20260806 16:16:32.029033 8192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:32.029107 8192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:16:32.029194 8181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:32.029912 8181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:32.035085 8181 catalog_manager.cc:1347] Generated new cluster ID: 15f3202ae4e3407db11589b901e18db4
I20260806 16:16:32.035154 8181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:32.048408 8181 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:32.048759 8181 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:32.054513 8181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f2ebd9416544cc98927546c2bd7a87: Generated new TSK 0
I20260806 16:16:32.055179 8181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:32.061133 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8098
2026-08-06T16:16:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:32 dist-test-slave-0b58 krb5kdc[8198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:32 dist-test-slave-0b58 krb5kdc[8198](info): set up 2 sockets
Aug 06 16:16:32 dist-test-slave-0b58 krb5kdc[8198](info): commencing operation
krb5kdc: starting...
W20260806 16:16:33.096480 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:33 dist-test-slave-0b58 krb5kdc[8198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:33.221618 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44509
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:33.332229 8212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:33.332566 8212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:33.332640 8212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:33.336613 8212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:33.336709 8212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:33.336745 8212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:33.336802 8212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:33.336839 8212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:33.342500 8212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:44509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:33.343653 8212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:33.344738 8212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:33.353284 8217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:33.353412 8212 server_base.cc:1034] running on GCE node
W20260806 16:16:33.353284 8218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:33.353313 8220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:33.354152 8212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:33.355296 8212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:33.356453 8212 hybrid_clock.cc:648] HybridClock initialized: now 1786032993356437 us; error 26 us; skew 500 ppm
Aug 06 16:16:33 dist-test-slave-0b58 krb5kdc[8198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:33.359246 8212 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:33.360162 8212 webserver.cc:460] Webserver started at http://127.18.90.62:36035/ using document root <none> and password file <none>
I20260806 16:16:33.360740 8212 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:33.360821 8212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:33.361069 8212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:33.363161 8212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "428d9fff3c6a4140b8fee4b136edf35d"
format_stamp: "Formatted at 2026-08-06 16:16:33 on dist-test-slave-0b58"
server_key: "4579c53288f68060a07b5ccb92c52aa7"
server_key_iv: "f4a7b70ab0303abb267223c031d60d26"
server_key_version: "encryptionkey@0"
I20260806 16:16:33.363732 8212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "428d9fff3c6a4140b8fee4b136edf35d"
format_stamp: "Formatted at 2026-08-06 16:16:33 on dist-test-slave-0b58"
server_key: "4579c53288f68060a07b5ccb92c52aa7"
server_key_iv: "f4a7b70ab0303abb267223c031d60d26"
server_key_version: "encryptionkey@0"
I20260806 16:16:33.367511 8212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:16:33.369946 8228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:33.371249 8212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:33.371384 8212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "428d9fff3c6a4140b8fee4b136edf35d"
format_stamp: "Formatted at 2026-08-06 16:16:33 on dist-test-slave-0b58"
server_key: "4579c53288f68060a07b5ccb92c52aa7"
server_key_iv: "f4a7b70ab0303abb267223c031d60d26"
server_key_version: "encryptionkey@0"
I20260806 16:16:33.371490 8212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:33.395542 8212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:33.398597 8212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:33.398839 8212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:33.406116 8212 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:44509
I20260806 16:16:33.406155 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:44509 every 8 connection(s)
I20260806 16:16:33.407133 8212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:33.408224 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8212
I20260806 16:16:33.408411 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:33.408717 18792 external_mini_cluster.cc:1428] Setting key 6f53ef18a2dcaa4a8a5176e1b8ef008d
I20260806 16:16:33.410303 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:33.415934 8282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d: Bootstrap starting.
Aug 06 16:16:33 dist-test-slave-0b58 krb5kdc[8198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:33.418495 8282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:33.419402 8282 log.cc:824] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:33.421440 8282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d: No bootstrap required, opened a new log
I20260806 16:16:33.423580 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:33.410160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39459 (local address 127.18.90.62:44509)
0806 16:16:33.410596 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:33.410611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:33.410648 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:16:33.411321 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:33.411337 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:33.411733 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:33.412060 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:33.412948 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:33.413958 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:33.414861 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:33.415351 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:33.415532 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:33.418353 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:33.418358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:33.418372 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:33.418399 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:33.420407 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:33.420904 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:33.420910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:33.420921 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:33.421027 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:33.421370 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:33.421373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:33.421374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:33.421859 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:33.422070 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:33.422393 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 16:16:33.422652 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260806 16:16:33.425298 8282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d9fff3c6a4140b8fee4b136edf35d" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44509 } }
I20260806 16:16:33.425593 8282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:33.425670 8282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428d9fff3c6a4140b8fee4b136edf35d, State: Initialized, Role: FOLLOWER
I20260806 16:16:33.426102 8282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d9fff3c6a4140b8fee4b136edf35d" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44509 } }
I20260806 16:16:33.426272 8282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:33.426358 8282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:33.426474 8282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:33.426734 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:33.427459 8282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d9fff3c6a4140b8fee4b136edf35d" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44509 } }
I20260806 16:16:33.427795 8282 leader_election.cc:302] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 428d9fff3c6a4140b8fee4b136edf35d; no voters:
I20260806 16:16:33.428140 8282 leader_election.cc:288] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:33.428210 8290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:33.428459 8290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [term 1 LEADER]: Becoming Leader. State: Replica: 428d9fff3c6a4140b8fee4b136edf35d, State: Running, Role: LEADER
I20260806 16:16:33.428762 8290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d9fff3c6a4140b8fee4b136edf35d" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44509 } }
I20260806 16:16:33.429507 8282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:33.430516 8292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "428d9fff3c6a4140b8fee4b136edf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d9fff3c6a4140b8fee4b136edf35d" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44509 } } }
I20260806 16:16:33.431006 8292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:33.430853 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 428d9fff3c6a4140b8fee4b136edf35d. Latest consensus state: current_term: 1 leader_uuid: "428d9fff3c6a4140b8fee4b136edf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d9fff3c6a4140b8fee4b136edf35d" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 44509 } } }
I20260806 16:16:33.431288 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:33.431562 8304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:33.435652 8304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:33.438745 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:33.429204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50963 (local address 127.18.90.62:44509)
0806 16:16:33.429307 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:33.429312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:33.429462 (+ 150us) server_negotiation.cc:407] Connection header received
0806 16:16:33.429646 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:33.429649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:33.429696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:33.429778 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:33.430783 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:33.431602 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:33.432770 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:33.433103 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:33.433238 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:33.434957 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:33.434961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:33.434963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:33.434990 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:33.436791 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:33.437353 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:33.437358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:33.437360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:33.437447 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:33.437812 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:33.437815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:33.437817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:33.438022 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:33.438167 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:33.438479 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 16:16:33.438574 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 16:16:33.442236 8304 catalog_manager.cc:1347] Generated new cluster ID: c73529b67c85403ebaa25cdfc81fa979
I20260806 16:16:33.442312 8304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:33.462695 8304 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:33.463696 8304 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:33.469695 8304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 428d9fff3c6a4140b8fee4b136edf35d: Generated new TSK 0
I20260806 16:16:33.470383 8304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:33.476641 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8212
W20260806 16:16:33.482679 8196 connection.cc:535] client connection to 127.18.90.62:44509 recv error: Network error: failed to read from TLS socket (remote: 127.18.90.62:44509): Connection reset by peer (error 104)
2026-08-06T16:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:33 dist-test-slave-0b58 krb5kdc[8314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:33 dist-test-slave-0b58 krb5kdc[8314](info): set up 2 sockets
Aug 06 16:16:33 dist-test-slave-0b58 krb5kdc[8314](info): commencing operation
krb5kdc: starting...
W20260806 16:16:34.520480 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:34 dist-test-slave-0b58 krb5kdc[8314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:34.642617 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:43727
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:43727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:34.756487 8327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:34.756757 8327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:34.756845 8327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:34.760838 8327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:34.760914 8327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:34.760950 8327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:34.760998 8327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:34.761035 8327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:34.766333 8327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:43727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:34.767396 8327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:34.768458 8327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:34.777030 8333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:34.777078 8327 server_base.cc:1034] running on GCE node
W20260806 16:16:34.777036 8332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:34.777030 8335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:34.777777 8327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:34.778872 8327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:34.780051 8327 hybrid_clock.cc:648] HybridClock initialized: now 1786032994780024 us; error 36 us; skew 500 ppm
Aug 06 16:16:34 dist-test-slave-0b58 krb5kdc[8314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:34.783106 8327 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:34.783938 8327 webserver.cc:460] Webserver started at http://127.18.90.62:34835/ using document root <none> and password file <none>
I20260806 16:16:34.784521 8327 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:34.784607 8327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:34.784817 8327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:34.786795 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "d7919b7b7b3b45168d6b25f63ddc5176"
format_stamp: "Formatted at 2026-08-06 16:16:34 on dist-test-slave-0b58"
server_key: "7210f3533c1b137a66828b34aa6817b1"
server_key_iv: "24145a182ede17611889a4adc9cd9321"
server_key_version: "encryptionkey@0"
I20260806 16:16:34.787346 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "d7919b7b7b3b45168d6b25f63ddc5176"
format_stamp: "Formatted at 2026-08-06 16:16:34 on dist-test-slave-0b58"
server_key: "7210f3533c1b137a66828b34aa6817b1"
server_key_iv: "24145a182ede17611889a4adc9cd9321"
server_key_version: "encryptionkey@0"
I20260806 16:16:34.790935 8327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 16:16:34.793336 8343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:34.794536 8327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:34.794662 8327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "d7919b7b7b3b45168d6b25f63ddc5176"
format_stamp: "Formatted at 2026-08-06 16:16:34 on dist-test-slave-0b58"
server_key: "7210f3533c1b137a66828b34aa6817b1"
server_key_iv: "24145a182ede17611889a4adc9cd9321"
server_key_version: "encryptionkey@0"
I20260806 16:16:34.794762 8327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:34.815217 8327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:34.818010 8327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:34.818208 8327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:34.824890 8327 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:43727
I20260806 16:16:34.824913 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:43727 every 8 connection(s)
I20260806 16:16:34.825801 8327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:34.828491 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:34.829239 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8327
I20260806 16:16:34.829383 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:34.829696 18792 external_mini_cluster.cc:1428] Setting key 583ad979163139504ca8a11e80423d9b
I20260806 16:16:34.834491 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176: Bootstrap starting.
I20260806 16:16:34.836783 8396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:34.837489 8396 log.cc:824] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:34.838550 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:34.831115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50321 (local address 127.18.90.62:43727)
0806 16:16:34.831551 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:34.831567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:34.831601 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:16:34.832231 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:34.832247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:34.832633 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:34.832950 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:34.833321 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:34.834241 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:34.835152 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:34.835590 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:34.835786 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:34.836693 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:34.836698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:34.836710 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:34.837140 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:34.837193 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:34.837215 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:34.837362 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:34.837523 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 16:16:34.837666 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260806 16:16:34.839668 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176: No bootstrap required, opened a new log
I20260806 16:16:34.841383 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:34.843537 8396 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43727 } }
I20260806 16:16:34.843863 8396 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:34.843931 8396 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7919b7b7b3b45168d6b25f63ddc5176, State: Initialized, Role: FOLLOWER
I20260806 16:16:34.844434 8396 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43727 } }
I20260806 16:16:34.844558 8396 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:34.844648 8396 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:34.844774 8396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:34.845835 8396 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43727 } }
I20260806 16:16:34.846184 8396 leader_election.cc:302] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7919b7b7b3b45168d6b25f63ddc5176; no voters:
I20260806 16:16:34.846495 8396 leader_election.cc:288] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:34.846563 8406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:34.846771 8406 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [term 1 LEADER]: Becoming Leader. State: Replica: d7919b7b7b3b45168d6b25f63ddc5176, State: Running, Role: LEADER
I20260806 16:16:34.847385 8406 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43727 } }
I20260806 16:16:34.847782 8396 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:34.848075 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:34.843266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60421 (local address 127.18.90.62:43727)
0806 16:16:34.843382 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:34.843386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:34.843554 (+ 168us) server_negotiation.cc:407] Connection header received
0806 16:16:34.843685 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:34.843687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:34.843737 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:34.843797 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:34.844128 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:34.844797 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:34.845974 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:34.846247 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:34.846359 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:34.847383 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:34.847387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:34.847390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:34.847401 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:34.847431 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:34.847453 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:34.847546 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:34.847734 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 16:16:34.847767 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:16:34.848625 8406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7919b7b7b3b45168d6b25f63ddc5176. Latest consensus state: current_term: 1 leader_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43727 } } }
I20260806 16:16:34.848762 8406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:34.849411 8407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7919b7b7b3b45168d6b25f63ddc5176" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 43727 } } }
I20260806 16:16:34.849525 8407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:34.849829 8415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:34.852780 8415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:34.857717 8415 catalog_manager.cc:1347] Generated new cluster ID: 2183437a62b74010a6c20d92a2ebfe4b
I20260806 16:16:34.857781 8415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:34.865937 8415 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:34.867055 8415 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:34.873606 8415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7919b7b7b3b45168d6b25f63ddc5176: Generated new TSK 0
I20260806 16:16:34.874328 8415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:34.885002 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8327
W20260806 16:16:34.890859 8312 connection.cc:535] client connection to 127.18.90.62:43727 recv error: Network error: failed to read from TLS socket (remote: 127.18.90.62:43727): Connection reset by peer (error 104)
2026-08-06T16:16:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:34 dist-test-slave-0b58 krb5kdc[8427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:34 dist-test-slave-0b58 krb5kdc[8427](info): set up 2 sockets
Aug 06 16:16:34 dist-test-slave-0b58 krb5kdc[8427](info): commencing operation
krb5kdc: starting...
W20260806 16:16:35.924929 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:36 dist-test-slave-0b58 krb5kdc[8427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:36.050822 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:40601
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:36.165057 8440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:36.165390 8440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:36.165495 8440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:36.169729 8440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:36.169827 8440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:36.169865 8440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:36.169921 8440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:36.169961 8440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:36.175871 8440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:40601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:36.177250 8440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:36.178311 8440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:36.187047 8445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:36.187050 8448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:36.187062 8446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:36.187422 8440 server_base.cc:1034] running on GCE node
I20260806 16:16:36.188037 8440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:36.189270 8440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:36.190496 8440 hybrid_clock.cc:648] HybridClock initialized: now 1786032996190480 us; error 26 us; skew 500 ppm
Aug 06 16:16:36 dist-test-slave-0b58 krb5kdc[8427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032996, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:36.193480 8440 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:36.194464 8440 webserver.cc:460] Webserver started at http://127.18.90.62:34139/ using document root <none> and password file <none>
I20260806 16:16:36.195086 8440 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:36.195182 8440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:36.195432 8440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:36.197878 8440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "470ca97cd91f47b199c764e0a12c1c34"
format_stamp: "Formatted at 2026-08-06 16:16:36 on dist-test-slave-0b58"
server_key: "217250b6f0cf34e66bd3a33dc0e47f65"
server_key_iv: "c12dac81b09a46ca29b5db4629509dec"
server_key_version: "encryptionkey@0"
I20260806 16:16:36.198490 8440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "470ca97cd91f47b199c764e0a12c1c34"
format_stamp: "Formatted at 2026-08-06 16:16:36 on dist-test-slave-0b58"
server_key: "217250b6f0cf34e66bd3a33dc0e47f65"
server_key_iv: "c12dac81b09a46ca29b5db4629509dec"
server_key_version: "encryptionkey@0"
I20260806 16:16:36.202666 8440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:16:36.205397 8455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:36.206777 8440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:16:36.206933 8440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "470ca97cd91f47b199c764e0a12c1c34"
format_stamp: "Formatted at 2026-08-06 16:16:36 on dist-test-slave-0b58"
server_key: "217250b6f0cf34e66bd3a33dc0e47f65"
server_key_iv: "c12dac81b09a46ca29b5db4629509dec"
server_key_version: "encryptionkey@0"
I20260806 16:16:36.207062 8440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:36.212179 8440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:36.215227 8440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:36.215440 8440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:36.223109 8440 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:40601
I20260806 16:16:36.223194 8509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:40601 every 8 connection(s)
I20260806 16:16:36.224656 8440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:36.227742 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8440
I20260806 16:16:36.227986 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:36.228363 18792 external_mini_cluster.cc:1428] Setting key 0b587a9cdae51ecc41f98917eace554f
I20260806 16:16:36.228586 8510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:36.234908 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:36.229823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47287 (local address 127.18.90.62:40601)
0806 16:16:36.230363 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:36.230386 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:36.230434 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:16:36.231900 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:36.231922 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:36.232400 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:36.232834 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:36.233646 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:36.233652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:36.233668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:36.233685 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:36.233739 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:36.233760 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:36.233927 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:36.234004 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 16:16:36.234176 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":158,"threads_started":1}
I20260806 16:16:36.236020 8510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34: Bootstrap starting.
I20260806 16:16:36.237865 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:36.238459 8510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:36.239266 8510 log.cc:824] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:36.241401 8510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34: No bootstrap required, opened a new log
I20260806 16:16:36.241814 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:36.240045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54013 (local address 127.18.90.62:40601)
0806 16:16:36.240156 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:36.240162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:36.240406 (+ 244us) server_negotiation.cc:407] Connection header received
0806 16:16:36.240531 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:36.240534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:36.240565 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:36.240648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:36.241371 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:36.241374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:36.241377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:36.241428 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:36.241452 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:36.241463 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:36.241560 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:36.241659 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 16:16:36.241698 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:16:36.245522 8510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470ca97cd91f47b199c764e0a12c1c34" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40601 } }
I20260806 16:16:36.245774 8510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:36.245852 8510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470ca97cd91f47b199c764e0a12c1c34, State: Initialized, Role: FOLLOWER
I20260806 16:16:36.246323 8510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470ca97cd91f47b199c764e0a12c1c34" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40601 } }
I20260806 16:16:36.246477 8510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:36.246564 8510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:36.246690 8510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:36.247692 8510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470ca97cd91f47b199c764e0a12c1c34" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40601 } }
I20260806 16:16:36.248052 8510 leader_election.cc:302] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 470ca97cd91f47b199c764e0a12c1c34; no voters:
I20260806 16:16:36.248478 8510 leader_election.cc:288] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:36.248615 8520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:36.248903 8520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [term 1 LEADER]: Becoming Leader. State: Replica: 470ca97cd91f47b199c764e0a12c1c34, State: Running, Role: LEADER
I20260806 16:16:36.249229 8520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470ca97cd91f47b199c764e0a12c1c34" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40601 } }
I20260806 16:16:36.249887 8510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:36.250432 8522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 470ca97cd91f47b199c764e0a12c1c34. Latest consensus state: current_term: 1 leader_uuid: "470ca97cd91f47b199c764e0a12c1c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470ca97cd91f47b199c764e0a12c1c34" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40601 } } }
I20260806 16:16:36.250561 8522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:36.250908 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "470ca97cd91f47b199c764e0a12c1c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470ca97cd91f47b199c764e0a12c1c34" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 40601 } } }
I20260806 16:16:36.251019 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:36.251398 8526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:36.254889 8526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:36.260671 8526 catalog_manager.cc:1347] Generated new cluster ID: f8585b0f8c1547ab907fa1f328f32d82
I20260806 16:16:36.260752 8526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:36.273974 8526 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:36.274302 8526 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:36.286746 8526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 470ca97cd91f47b199c764e0a12c1c34: Generated new TSK 0
I20260806 16:16:36.287497 8526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:36.299860 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8440
2026-08-06T16:16:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:36 dist-test-slave-0b58 krb5kdc[8542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:36 dist-test-slave-0b58 krb5kdc[8542](info): set up 2 sockets
Aug 06 16:16:36 dist-test-slave-0b58 krb5kdc[8542](info): commencing operation
krb5kdc: starting...
W20260806 16:16:37.336796 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:37 dist-test-slave-0b58 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:37.463001 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42697
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:37.575167 8556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:37.575480 8556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:37.575553 8556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:37.579516 8556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:37.579608 8556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:37.579646 8556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:37.579700 8556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:37.579741 8556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:37.584924 8556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:37.585978 8556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:37.587021 8556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:37.595268 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:37.595276 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:37.595420 8556 server_base.cc:1034] running on GCE node
W20260806 16:16:37.595610 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:37.596160 8556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:37.597262 8556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:37.598445 8556 hybrid_clock.cc:648] HybridClock initialized: now 1786032997598431 us; error 25 us; skew 500 ppm
Aug 06 16:16:37 dist-test-slave-0b58 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032997, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:37.605147 8556 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:37.606060 8556 webserver.cc:460] Webserver started at http://127.18.90.62:37455/ using document root <none> and password file <none>
I20260806 16:16:37.606637 8556 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:37.606725 8556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:37.606953 8556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:37.609243 8556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "da0f0e5ca26649058ab8eab50451615f"
format_stamp: "Formatted at 2026-08-06 16:16:37 on dist-test-slave-0b58"
server_key: "b3d44cee03659399b88e845a65df5063"
server_key_iv: "40e1a773d1aebd124ce4b9a6292ff17b"
server_key_version: "encryptionkey@0"
I20260806 16:16:37.609905 8556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "da0f0e5ca26649058ab8eab50451615f"
format_stamp: "Formatted at 2026-08-06 16:16:37 on dist-test-slave-0b58"
server_key: "b3d44cee03659399b88e845a65df5063"
server_key_iv: "40e1a773d1aebd124ce4b9a6292ff17b"
server_key_version: "encryptionkey@0"
I20260806 16:16:37.613642 8556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:16:37.615912 8571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:37.617134 8556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:37.617283 8556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "da0f0e5ca26649058ab8eab50451615f"
format_stamp: "Formatted at 2026-08-06 16:16:37 on dist-test-slave-0b58"
server_key: "b3d44cee03659399b88e845a65df5063"
server_key_iv: "40e1a773d1aebd124ce4b9a6292ff17b"
server_key_version: "encryptionkey@0"
I20260806 16:16:37.617393 8556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:37.629660 8556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:37.632596 8556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:37.632802 8556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:37.639654 8556 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:42697
I20260806 16:16:37.639690 8625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:42697 every 8 connection(s)
I20260806 16:16:37.640638 8556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:37.643405 8626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:37.648926 8626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f: Bootstrap starting.
I20260806 16:16:37.649755 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8556
I20260806 16:16:37.649895 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:37.650179 18792 external_mini_cluster.cc:1428] Setting key 99fe66c4294fb9b392a4ae704ff57a49
I20260806 16:16:37.651839 8626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:37.652751 8626 log.cc:824] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:37.654940 8626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f: No bootstrap required, opened a new log
I20260806 16:16:37.659045 8626 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0f0e5ca26649058ab8eab50451615f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42697 } }
I20260806 16:16:37.659314 8626 raft_consensus.cc:374] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 06 16:16:37 dist-test-slave-0b58 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:37.659397 8626 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0f0e5ca26649058ab8eab50451615f, State: Initialized, Role: FOLLOWER
I20260806 16:16:37.659868 8626 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0f0e5ca26649058ab8eab50451615f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42697 } }
I20260806 16:16:37.660024 8626 raft_consensus.cc:388] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:37.660099 8626 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:37.660200 8626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:37.661527 8626 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0f0e5ca26649058ab8eab50451615f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42697 } }
I20260806 16:16:37.661911 8626 leader_election.cc:302] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da0f0e5ca26649058ab8eab50451615f; no voters:
I20260806 16:16:37.662267 8626 leader_election.cc:288] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:37.662421 8631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:37.662669 8631 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [term 1 LEADER]: Becoming Leader. State: Replica: da0f0e5ca26649058ab8eab50451615f, State: Running, Role: LEADER
I20260806 16:16:37.663029 8631 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0f0e5ca26649058ab8eab50451615f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42697 } }
I20260806 16:16:37.663646 8626 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:37.664865 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0f0e5ca26649058ab8eab50451615f. Latest consensus state: current_term: 1 leader_uuid: "da0f0e5ca26649058ab8eab50451615f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0f0e5ca26649058ab8eab50451615f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42697 } } }
I20260806 16:16:37.665040 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:37.665959 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:37.651392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52929 (local address 127.18.90.62:42697)
0806 16:16:37.651870 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:37.651891 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:37.651934 (+ 43us) server_negotiation.cc:407] Connection header received
0806 16:16:37.652845 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:37.652870 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:37.653292 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:37.653634 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:37.654272 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:37.655678 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:37.656584 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:37.657293 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:37.657469 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:37.660088 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:37.660094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:37.660112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:37.660148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:37.662190 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:37.662684 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:37.662689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:37.662695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:37.662833 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:37.663158 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:37.663162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:37.663163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:37.663664 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:37.663866 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:37.664558 (+ 692us) server_negotiation.cc:299] Negotiation successful
0806 16:16:37.664929 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":113,"threads_started":1}
I20260806 16:16:37.666370 8632 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da0f0e5ca26649058ab8eab50451615f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0f0e5ca26649058ab8eab50451615f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42697 } } }
I20260806 16:16:37.666188 8640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:37.666489 8632 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:37.669464 8640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:37.669479 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:37.675220 8640 catalog_manager.cc:1347] Generated new cluster ID: fdab783fb15042e2969ab50077b16fe9
I20260806 16:16:37.675280 8640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:37.680886 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:37.671809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37097 (local address 127.18.90.62:42697)
0806 16:16:37.671925 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:37.671929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:37.672185 (+ 256us) server_negotiation.cc:407] Connection header received
0806 16:16:37.672330 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:37.672333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:37.672384 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:37.672466 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:37.673301 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:37.674082 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:37.675274 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:37.675632 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:37.675706 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:37.677471 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:37.677474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:37.677477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:37.677502 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:37.679090 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:37.679595 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:37.679598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:37.679600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:37.679682 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:37.679962 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:37.679965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:37.679966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:37.680149 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:37.680284 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:37.680619 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 16:16:37.680735 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 16:16:37.683987 8640 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:37.685252 8640 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:37.697319 8640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0f0e5ca26649058ab8eab50451615f: Generated new TSK 0
I20260806 16:16:37.698199 8640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:37.702525 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8556
2026-08-06T16:16:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:37 dist-test-slave-0b58 krb5kdc[8657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:37 dist-test-slave-0b58 krb5kdc[8657](info): set up 2 sockets
Aug 06 16:16:37 dist-test-slave-0b58 krb5kdc[8657](info): commencing operation
krb5kdc: starting...
W20260806 16:16:38.732713 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:38 dist-test-slave-0b58 krb5kdc[8657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:38.859702 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42465
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:38.973497 8670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:38.973757 8670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:38.973815 8670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:38.977685 8670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:38.977741 8670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:38.977761 8670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:38.977775 8670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:38.977792 8670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:38.982887 8670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:38.983937 8670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:38.984949 8670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:38.993654 8679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:38.993654 8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:38.993772 8677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:38.993963 8670 server_base.cc:1034] running on GCE node
I20260806 16:16:38.994510 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:38.995507 8670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:38.996660 8670 hybrid_clock.cc:648] HybridClock initialized: now 1786032998996635 us; error 42 us; skew 500 ppm
Aug 06 16:16:38 dist-test-slave-0b58 krb5kdc[8657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:38.999303 8670 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:39.000192 8670 webserver.cc:460] Webserver started at http://127.18.90.62:38745/ using document root <none> and password file <none>
I20260806 16:16:39.000792 8670 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:39.000880 8670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:39.001120 8670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:39.003136 8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "a57f059b4eb1433f91cf03b4585312c5"
format_stamp: "Formatted at 2026-08-06 16:16:38 on dist-test-slave-0b58"
server_key: "33af97b099fc7d7a010c56d1b18e47ce"
server_key_iv: "9154b19c7a32fa931ea24f42e4dc8661"
server_key_version: "encryptionkey@0"
I20260806 16:16:39.003743 8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "a57f059b4eb1433f91cf03b4585312c5"
format_stamp: "Formatted at 2026-08-06 16:16:38 on dist-test-slave-0b58"
server_key: "33af97b099fc7d7a010c56d1b18e47ce"
server_key_iv: "9154b19c7a32fa931ea24f42e4dc8661"
server_key_version: "encryptionkey@0"
I20260806 16:16:39.007530 8670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:16:39.009974 8686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:39.011183 8670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:39.011334 8670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "a57f059b4eb1433f91cf03b4585312c5"
format_stamp: "Formatted at 2026-08-06 16:16:38 on dist-test-slave-0b58"
server_key: "33af97b099fc7d7a010c56d1b18e47ce"
server_key_iv: "9154b19c7a32fa931ea24f42e4dc8661"
server_key_version: "encryptionkey@0"
I20260806 16:16:39.011439 8670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:39.036767 8670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:39.039685 8670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:39.039901 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:39.046790 8670 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:42465
I20260806 16:16:39.046805 8738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:42465 every 8 connection(s)
I20260806 16:16:39.047729 8670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:39.050482 8739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:39.055640 8739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5: Bootstrap starting.
I20260806 16:16:39.056722 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8670
I20260806 16:16:39.056859 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:39.057130 18792 external_mini_cluster.cc:1428] Setting key 1985bd9ab3d657502b267cfb9ba46de4
I20260806 16:16:39.058444 8739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:39.059154 8739 log.cc:824] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:39.061254 8739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5: No bootstrap required, opened a new log
I20260806 16:16:39.065240 8739 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57f059b4eb1433f91cf03b4585312c5" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42465 } }
I20260806 16:16:39.065462 8739 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:39.065548 8739 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57f059b4eb1433f91cf03b4585312c5, State: Initialized, Role: FOLLOWER
I20260806 16:16:39.065677 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:39.058196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41377 (local address 127.18.90.62:42465)
0806 16:16:39.058600 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:39.058615 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:39.058650 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:16:39.059247 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:39.059300 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:39.059712 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:39.060065 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:39.060449 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:39.061405 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:39.062560 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:39.063008 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:39.063149 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:39.063984 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:39.063989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:39.064000 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:39.064401 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:39.064454 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:39.064475 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:39.064547 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:39.064697 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 16:16:39.064907 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
I20260806 16:16:39.066083 8739 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57f059b4eb1433f91cf03b4585312c5" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42465 } }
I20260806 16:16:39.066284 8739 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:39.066350 8739 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:39.066442 8739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:39.067337 8739 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57f059b4eb1433f91cf03b4585312c5" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42465 } }
I20260806 16:16:39.067714 8739 leader_election.cc:302] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57f059b4eb1433f91cf03b4585312c5; no voters:
I20260806 16:16:39.068068 8739 leader_election.cc:288] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:39.068161 8744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:39.068390 8744 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [term 1 LEADER]: Becoming Leader. State: Replica: a57f059b4eb1433f91cf03b4585312c5, State: Running, Role: LEADER
I20260806 16:16:39.068480 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:39.068696 8744 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57f059b4eb1433f91cf03b4585312c5" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42465 } }
I20260806 16:16:39.069135 8739 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:39.069922 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57f059b4eb1433f91cf03b4585312c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57f059b4eb1433f91cf03b4585312c5" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42465 } } }
I20260806 16:16:39.070029 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:39.070232 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57f059b4eb1433f91cf03b4585312c5. Latest consensus state: current_term: 1 leader_uuid: "a57f059b4eb1433f91cf03b4585312c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57f059b4eb1433f91cf03b4585312c5" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42465 } } }
I20260806 16:16:39.070309 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:39.071026 8757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:39.074107 8757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:39.075565 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:39.070882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56319 (local address 127.18.90.62:42465)
0806 16:16:39.070980 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:39.070984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:39.071211 (+ 227us) server_negotiation.cc:407] Connection header received
0806 16:16:39.071448 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:39.071451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:39.071493 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:39.071543 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:39.071961 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:39.072750 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:39.073815 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:39.074141 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:39.074246 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:39.075118 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:39.075122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:39.075125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:39.075137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:39.075166 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:39.075180 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:39.075282 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:39.075383 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 16:16:39.075433 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 16:16:39.079770 8757 catalog_manager.cc:1347] Generated new cluster ID: 97093f38607348ad867d747ce84820ba
I20260806 16:16:39.079833 8757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:39.101377 8757 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:39.102622 8757 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:39.115702 8757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a57f059b4eb1433f91cf03b4585312c5: Generated new TSK 0
I20260806 16:16:39.116587 8757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:39.120782 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8670
2026-08-06T16:16:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:39 dist-test-slave-0b58 krb5kdc[8770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:39 dist-test-slave-0b58 krb5kdc[8770](info): set up 2 sockets
Aug 06 16:16:39 dist-test-slave-0b58 krb5kdc[8770](info): commencing operation
krb5kdc: starting...
W20260806 16:16:40.148805 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:40 dist-test-slave-0b58 krb5kdc[8770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:40.288763 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33489
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:40.402453 8784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:40.402786 8784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:40.402863 8784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:40.407006 8784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:40.407106 8784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:40.407145 8784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:40.407200 8784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:40.407238 8784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:40.412719 8784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:40.413877 8784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:40.414943 8784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:40.423651 8790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:40.423667 8789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:40.423651 8792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:40.423784 8784 server_base.cc:1034] running on GCE node
I20260806 16:16:40.424574 8784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:40.425781 8784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:40.426942 8784 hybrid_clock.cc:648] HybridClock initialized: now 1786033000426922 us; error 33 us; skew 500 ppm
Aug 06 16:16:40 dist-test-slave-0b58 krb5kdc[8770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033000, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:40.429710 8784 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:40.430725 8784 webserver.cc:460] Webserver started at http://127.18.90.62:42253/ using document root <none> and password file <none>
I20260806 16:16:40.431389 8784 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:40.431457 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:40.431684 8784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:40.433770 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "871c3a84f82c46599830b5d2acbc1c6f"
format_stamp: "Formatted at 2026-08-06 16:16:40 on dist-test-slave-0b58"
server_key: "a782306e9e92482d8cd9ed423154f89e"
server_key_iv: "3f824404d011387010c37d63c001cc88"
server_key_version: "encryptionkey@0"
I20260806 16:16:40.434345 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "871c3a84f82c46599830b5d2acbc1c6f"
format_stamp: "Formatted at 2026-08-06 16:16:40 on dist-test-slave-0b58"
server_key: "a782306e9e92482d8cd9ed423154f89e"
server_key_iv: "3f824404d011387010c37d63c001cc88"
server_key_version: "encryptionkey@0"
I20260806 16:16:40.438235 8784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:16:40.440645 8799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:40.441924 8784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:40.442073 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "871c3a84f82c46599830b5d2acbc1c6f"
format_stamp: "Formatted at 2026-08-06 16:16:40 on dist-test-slave-0b58"
server_key: "a782306e9e92482d8cd9ed423154f89e"
server_key_iv: "3f824404d011387010c37d63c001cc88"
server_key_version: "encryptionkey@0"
I20260806 16:16:40.442207 8784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:40.446852 8784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:40.449833 8784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:40.450029 8784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:40.456840 8784 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:33489
I20260806 16:16:40.456915 8853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:33489 every 8 connection(s)
I20260806 16:16:40.457844 8784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:40.460633 8854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:40.465691 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8784
I20260806 16:16:40.465850 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:40.466220 18792 external_mini_cluster.cc:1428] Setting key 8da81a44b4b86207a6f3c7681b7ed2b4
I20260806 16:16:40.466235 8854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f: Bootstrap starting.
I20260806 16:16:40.468983 8854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:40.469936 8854 log.cc:824] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:40.471866 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:40.467607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35741 (local address 127.18.90.62:33489)
0806 16:16:40.468049 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:40.468072 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:40.468114 (+ 42us) server_negotiation.cc:407] Connection header received
0806 16:16:40.469008 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:40.469034 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:40.469452 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:40.469831 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:40.470622 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:40.470626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:40.470637 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:40.470646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:40.470693 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:40.470711 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:40.470837 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:40.470982 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 16:16:40.471149 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20260806 16:16:40.472421 8854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f: No bootstrap required, opened a new log
I20260806 16:16:40.474736 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:40.476438 8854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871c3a84f82c46599830b5d2acbc1c6f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33489 } }
I20260806 16:16:40.476696 8854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:16:40.476776 8854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871c3a84f82c46599830b5d2acbc1c6f, State: Initialized, Role: FOLLOWER
I20260806 16:16:40.477315 8854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871c3a84f82c46599830b5d2acbc1c6f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33489 } }
I20260806 16:16:40.477520 8854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:16:40.477605 8854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:16:40.477722 8854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:40.478556 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:40.476962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53195 (local address 127.18.90.62:33489)
0806 16:16:40.477076 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:40.477082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:40.477190 (+ 108us) server_negotiation.cc:407] Connection header received
0806 16:16:40.477307 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:40.477309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:40.477339 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:40.477402 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:40.478185 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:40.478187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:40.478189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:16:40.478199 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:40.478223 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:16:40.478233 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:40.478302 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:40.478398 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 16:16:40.478428 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:16:40.478957 8854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871c3a84f82c46599830b5d2acbc1c6f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33489 } }
I20260806 16:16:40.479308 8854 leader_election.cc:302] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 871c3a84f82c46599830b5d2acbc1c6f; no voters:
I20260806 16:16:40.479686 8854 leader_election.cc:288] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:16:40.479775 8865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:40.480017 8865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [term 1 LEADER]: Becoming Leader. State: Replica: 871c3a84f82c46599830b5d2acbc1c6f, State: Running, Role: LEADER
I20260806 16:16:40.480420 8865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871c3a84f82c46599830b5d2acbc1c6f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33489 } }
I20260806 16:16:40.481062 8854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:40.482124 8866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "871c3a84f82c46599830b5d2acbc1c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871c3a84f82c46599830b5d2acbc1c6f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33489 } } }
I20260806 16:16:40.482259 8866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:40.482646 8874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:40.482995 8867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871c3a84f82c46599830b5d2acbc1c6f. Latest consensus state: current_term: 1 leader_uuid: "871c3a84f82c46599830b5d2acbc1c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871c3a84f82c46599830b5d2acbc1c6f" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33489 } } }
I20260806 16:16:40.483088 8867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:40.485430 8874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:40.490962 8874 catalog_manager.cc:1347] Generated new cluster ID: 507507e199954609a376983cd47164df
I20260806 16:16:40.491042 8874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:40.518554 8874 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:40.519001 8874 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:40.528576 8874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 871c3a84f82c46599830b5d2acbc1c6f: Generated new TSK 0
I20260806 16:16:40.529561 8874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:16:40.535188 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8784
2026-08-06T16:16:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1415 ms)
[----------] 18 tests from AuthTokenIssuingTest (25361 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:40 dist-test-slave-0b58 krb5kdc[8886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:40 dist-test-slave-0b58 krb5kdc[8886](info): set up 2 sockets
Aug 06 16:16:40 dist-test-slave-0b58 krb5kdc[8886](info): commencing operation
krb5kdc: starting...
W20260806 16:16:41.565147 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:41 dist-test-slave-0b58 krb5kdc[8886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:41.691744 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33735
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33735,127.18.90.61:33537,127.18.90.60:39825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:41.805027 8901 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:16:41.805402 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:41.805478 8901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:41.805560 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:41.809657 8901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:41.809760 8901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:41.809801 8901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:41.809840 8901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:41.809877 8901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:41.815325 8901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33735,127.18.90.61:33537,127.18.90.60:39825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:41.816561 8901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:41.817633 8901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:41.826519 8906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:41.826501 8907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:41.826568 8901 server_base.cc:1034] running on GCE node
W20260806 16:16:41.826514 8909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:41.827385 8901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:41.828608 8901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:41.829795 8901 hybrid_clock.cc:648] HybridClock initialized: now 1786033001829779 us; error 28 us; skew 500 ppm
Aug 06 16:16:41 dist-test-slave-0b58 krb5kdc[8886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:41.832547 8901 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:41.833488 8901 webserver.cc:460] Webserver started at http://127.18.90.62:41429/ using document root <none> and password file <none>
I20260806 16:16:41.834249 8901 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:41.834349 8901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:41.834566 8901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:41.836669 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "8e62684db19d4ae19856bc821c264c81"
format_stamp: "Formatted at 2026-08-06 16:16:41 on dist-test-slave-0b58"
server_key: "f7b15df9ccce2c84785048574d501082"
server_key_iv: "18abe69166c45ad3f98be68de48f1c08"
server_key_version: "encryptionkey@0"
I20260806 16:16:41.837203 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "8e62684db19d4ae19856bc821c264c81"
format_stamp: "Formatted at 2026-08-06 16:16:41 on dist-test-slave-0b58"
server_key: "f7b15df9ccce2c84785048574d501082"
server_key_iv: "18abe69166c45ad3f98be68de48f1c08"
server_key_version: "encryptionkey@0"
I20260806 16:16:41.841056 8901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:16:41.843478 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:41.844875 8901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:41.845000 8901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "8e62684db19d4ae19856bc821c264c81"
format_stamp: "Formatted at 2026-08-06 16:16:41 on dist-test-slave-0b58"
server_key: "f7b15df9ccce2c84785048574d501082"
server_key_iv: "18abe69166c45ad3f98be68de48f1c08"
server_key_version: "encryptionkey@0"
I20260806 16:16:41.845108 8901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:41.863198 8901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:41.866189 8901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:41.866411 8901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:41.873710 8901 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:33735
I20260806 16:16:41.874028 8969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:33735 every 8 connection(s)
I20260806 16:16:41.874794 8901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:41.877717 8970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:41.878885 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8901
I20260806 16:16:41.879046 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:41.879379 18792 external_mini_cluster.cc:1428] Setting key dd9b77d3e6e406ae527a627d677a3aa8
I20260806 16:16:41.882869 8970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.90.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.61@KRBTEST.COM" created.
Aug 06 16:16:41 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:41.899617 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:41.885485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.18.90.62:33735)
0806 16:16:41.886090 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:41.886108 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:41.886315 (+ 207us) server_negotiation.cc:407] Connection header received
0806 16:16:41.887759 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:41.887763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:41.887945 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:41.888097 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:41.889162 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:41.890200 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:41.891426 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:41.891907 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:41.892102 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:41.894546 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:41.894555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:41.894569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:41.894600 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:41.896496 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:41.897103 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:41.897108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:41.897109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:41.897220 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:41.897609 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:41.897611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:41.897612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:41.897999 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:41.898103 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:41.898576 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 16:16:41.898799 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":477,"spinlock_wait_cycles":59136,"thread_start_us":162,"threads_started":1}
I20260806 16:16:41.899621 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:41.885485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33735 (local address 127.0.0.1:40360)
0806 16:16:41.886103 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:41.886190 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:41.887427 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:41.888167 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:41.888177 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:41.888678 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:41.889020 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:41.889036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:41.890336 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:41.890338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:41.891270 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:41.891283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:41.892082 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:41.892085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:41.892179 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:41.892937 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:41.892953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:41.894384 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:41.896608 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:41.896621 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:41.896635 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:41.896989 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:41.897322 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:41.897328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:41.897331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:41.897501 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:41.898123 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:41.898130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:41.898313 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 16:16:41.898513 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 16:16:41.898891 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":122,"thread_start_us":162,"threads_started":1}
I20260806 16:16:41.902112 8970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:41.903131 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:41.902587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57350)
0806 16:16:41.902993 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:41.903064 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
W20260806 16:16:41.903404 8917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
I20260806 16:16:41.904582 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:41.904245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57364)
0806 16:16:41.904384 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:41.904493 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 16:16:41.904764 8970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:33537: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab.
Entry for principal kudu/127.18.90.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab.
I20260806 16:16:41.908831 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:33537
--webserver_interface=127.18.90.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab
--principal=kudu/127.18.90.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33735,127.18.90.61:33537,127.18.90.60:39825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:16:41.954007 8970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } attempt: 1
I20260806 16:16:41.954883 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:41.954588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57380)
0806 16:16:41.954746 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:41.954807 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 16:16:41.955592 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:41.955378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57390)
0806 16:16:41.955485 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:41.955536 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:16:41.955706 8970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:33537: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
I20260806 16:16:42.023873 8970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } attempt: 2
I20260806 16:16:42.024744 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.024446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57398)
0806 16:16:42.024599 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.024663 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 16:16:42.025441 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.025222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57406)
0806 16:16:42.025331 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.025383 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 16:16:42.025370 8978 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:16:42.025592 8970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:33537: Network error: Client connection negotiation failed: client connection to 127.18.90.61:33537: connect: Connection refused (error 111)
W20260806 16:16:42.025709 8978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:42.025766 8978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:42.025839 8978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:42.029845 8978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:42.029917 8978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:42.029963 8978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:42.030014 8978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:42.030051 8978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:42.035398 8978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33735,127.18.90.61:33537,127.18.90.60:39825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab
--principal=kudu/127.18.90.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:33537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.90.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:42.036571 8978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:42.037592 8978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:42.046094 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:42.046144 8983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:42.046097 8984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:42.046464 8978 server_base.cc:1034] running on GCE node
I20260806 16:16:42.046985 8978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:42.048055 8978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:42.049269 8978 hybrid_clock.cc:648] HybridClock initialized: now 1786033002049247 us; error 59 us; skew 500 ppm
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:42.051944 8978 init.cc:377] Logged in from keytab as kudu/127.18.90.61@KRBTEST.COM (short username kudu)
I20260806 16:16:42.052837 8978 webserver.cc:460] Webserver started at http://127.18.90.61:34781/ using document root <none> and password file <none>
I20260806 16:16:42.053346 8978 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:42.053404 8978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:42.053565 8978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:42.055521 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data/instance:
uuid: "4ff548bae76f44a5ba0289326f88232a"
format_stamp: "Formatted at 2026-08-06 16:16:42 on dist-test-slave-0b58"
server_key: "a3529236d0bffb0c28f05927c19dc355"
server_key_iv: "b825ece8be77848f31e3a69dffd28aa2"
server_key_version: "encryptionkey@0"
I20260806 16:16:42.056038 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal/instance:
uuid: "4ff548bae76f44a5ba0289326f88232a"
format_stamp: "Formatted at 2026-08-06 16:16:42 on dist-test-slave-0b58"
server_key: "a3529236d0bffb0c28f05927c19dc355"
server_key_iv: "b825ece8be77848f31e3a69dffd28aa2"
server_key_version: "encryptionkey@0"
I20260806 16:16:42.059680 8978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:16:42.062059 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:42.063277 8978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:16:42.063416 8978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal
uuid: "4ff548bae76f44a5ba0289326f88232a"
format_stamp: "Formatted at 2026-08-06 16:16:42 on dist-test-slave-0b58"
server_key: "a3529236d0bffb0c28f05927c19dc355"
server_key_iv: "b825ece8be77848f31e3a69dffd28aa2"
server_key_version: "encryptionkey@0"
I20260806 16:16:42.063529 8978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:42.101292 8978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:42.104302 8978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:42.104521 8978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:42.111792 8978 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.61:33537
I20260806 16:16:42.112241 9047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.61:33537 every 8 connection(s)
I20260806 16:16:42.113013 8978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
I20260806 16:16:42.115724 9048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:42.116545 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 8978
I20260806 16:16:42.116670 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-1/wal/instance
I20260806 16:16:42.116736 8970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } attempt: 3
I20260806 16:16:42.117007 18792 external_mini_cluster.cc:1428] Setting key 8978b81cfa95d12602da730debb7e97f
I20260806 16:16:42.121655 9048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.60@KRBTEST.COM" created.
I20260806 16:16:42.130997 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.117329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57414)
0806 16:16:42.117547 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.117564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.117676 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.120859 (+ 3183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.120862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.121072 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.121210 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.121219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.122634 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.122637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.123190 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.123199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.124187 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.124191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.124370 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.124943 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.124960 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.126601 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.128950 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.128953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.128955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.129239 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.129669 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.129672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.129674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.129779 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.130492 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.130495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.130580 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.130666 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.130778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:42.132575 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.118066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.18.90.61:33537)
0806 16:16:42.118483 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.118501 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.118549 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:16:42.119976 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.120004 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.120509 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.120920 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.121381 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.122506 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.123328 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.124038 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.124664 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.126786 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.126791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.126806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.126832 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.128839 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.129377 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.129385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.129392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.129554 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.129882 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.129885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.129886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.130389 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.130496 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.130879 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.131488 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":138,"threads_started":1}
I20260806 16:16:42.134824 8970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:42.135635 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.135344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33206)
0806 16:16:42.135476 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.135540 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 16:16:42.136580 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.123949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.18.90.62:33735)
0806 16:16:42.124095 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.124101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.124596 (+ 495us) server_negotiation.cc:407] Connection header received
0806 16:16:42.124814 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.124818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.124870 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.124932 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.126154 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.127015 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.128426 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.128825 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.128975 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.131989 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.131996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.132002 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.132035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.134593 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.135143 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.135149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.135154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.135252 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.135577 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.135580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.135582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.135829 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.135949 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.136320 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.136434 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:16:42.136770 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.136117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33218)
0806 16:16:42.136522 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.136586 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":123,"threads_started":1}
I20260806 16:16:42.136610 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.123890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33735 (local address 127.0.0.1:40362)
0806 16:16:42.124441 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.124506 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.124691 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.125133 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.125140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.125631 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.125963 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.125979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.127195 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.127202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.128247 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.128300 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.128966 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.128969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.129095 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.129694 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.129712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.131834 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.134709 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.134719 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.134724 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.135019 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.135365 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.135371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.135374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.135474 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.135948 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.135954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.136073 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.136214 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.136435 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":139,"threads_started":1}
W20260806 16:16:42.136905 8970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:39825: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
I20260806 16:16:42.138057 9048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:42.147771 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.138536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57422)
0806 16:16:42.138644 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.138660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.138732 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.139067 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.139069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.139208 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.139296 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.139311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.140402 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.140405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.140909 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.140916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.141516 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.141519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.141592 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.142024 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.142035 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.144576 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.146309 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.146313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.146315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.146505 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.146787 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.146790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.146791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.146872 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.147247 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.147250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.147341 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.147442 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.147591 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 16:16:42.147778 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.138682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.18.90.61:33537)
0806 16:16:42.138824 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.138830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.138848 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:16:42.138890 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.138896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.138946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.139033 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.139416 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.140286 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.141030 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.141385 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.141530 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.144721 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.144724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.144726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.144746 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.146180 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.146602 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.146607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.146608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.146682 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.146961 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.146966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.146967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.147139 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.147247 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.147501 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.147652 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":48}
I20260806 16:16:42.149112 9048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:42.149830 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.149574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33234)
0806 16:16:42.149691 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.149766 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 16:16:42.150077 8996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab.
Entry for principal kudu/127.18.90.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab.
I20260806 16:16:42.151368 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:39825
--webserver_interface=127.18.90.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab
--principal=kudu/127.18.90.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:33735,127.18.90.61:33537,127.18.90.60:39825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:16:42.152547 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.150762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33238)
0806 16:16:42.152403 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.152465 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1552}
W20260806 16:16:42.152761 9048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:39825: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
I20260806 16:16:42.168085 8970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } attempt: 1
I20260806 16:16:42.169045 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.168674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33254)
0806 16:16:42.168844 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.168920 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 16:16:42.169806 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.169568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33266)
0806 16:16:42.169683 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.169739 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 16:16:42.169957 8970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:39825: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
I20260806 16:16:42.202112 9048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } attempt: 1
I20260806 16:16:42.203073 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.202751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33274)
0806 16:16:42.202908 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.202981 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 16:16:42.203866 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.203656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33284)
0806 16:16:42.203758 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.203804 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:16:42.204041 9048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:39825: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
I20260806 16:16:42.245148 8970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } attempt: 2
I20260806 16:16:42.246062 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.245676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33296)
0806 16:16:42.245886 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.245963 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 16:16:42.246886 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.246626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33308)
0806 16:16:42.246742 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.246804 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 16:16:42.247089 8970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:39825: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
W20260806 16:16:42.269013 9056 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:16:42.269301 9056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:42.269326 9056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:42.269371 9056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:16:42.272207 9048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } attempt: 2
I20260806 16:16:42.273088 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.272801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33324)
0806 16:16:42.272952 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.273014 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 16:16:42.273659 9056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:42.273739 9056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:42.273782 9056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:42.273818 9056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:42.273856 9056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:42.273849 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.273587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33328)
0806 16:16:42.273691 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.273742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:16:42.273981 9048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:39825: Network error: Client connection negotiation failed: client connection to 127.18.90.60:39825: connect: Connection refused (error 111)
I20260806 16:16:42.279155 9056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:33735,127.18.90.61:33537,127.18.90.60:39825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab
--principal=kudu/127.18.90.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:39825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.90.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:42.280373 9056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:42.281406 9056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:42.289670 9064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:42.289683 9061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:42.289708 9062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:42.289933 9056 server_base.cc:1034] running on GCE node
I20260806 16:16:42.290506 9056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:42.291793 9056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:42.292963 9056 hybrid_clock.cc:648] HybridClock initialized: now 1786033002292945 us; error 27 us; skew 500 ppm
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:42.295656 9056 init.cc:377] Logged in from keytab as kudu/127.18.90.60@KRBTEST.COM (short username kudu)
I20260806 16:16:42.296530 9056 webserver.cc:460] Webserver started at http://127.18.90.60:44877/ using document root <none> and password file <none>
I20260806 16:16:42.297036 9056 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:42.297092 9056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:42.297276 9056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:42.299263 9056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data/instance:
uuid: "5420cf263269416e82fa37b3b0d984c1"
format_stamp: "Formatted at 2026-08-06 16:16:42 on dist-test-slave-0b58"
server_key: "a516af905ed656df33d955b3709fd199"
server_key_iv: "e24c6be09a51753bac8c037f4c2dbbd6"
server_key_version: "encryptionkey@0"
I20260806 16:16:42.299803 9056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal/instance:
uuid: "5420cf263269416e82fa37b3b0d984c1"
format_stamp: "Formatted at 2026-08-06 16:16:42 on dist-test-slave-0b58"
server_key: "a516af905ed656df33d955b3709fd199"
server_key_iv: "e24c6be09a51753bac8c037f4c2dbbd6"
server_key_version: "encryptionkey@0"
I20260806 16:16:42.303463 9056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:16:42.305912 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:42.307160 9056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:42.307305 9056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal
uuid: "5420cf263269416e82fa37b3b0d984c1"
format_stamp: "Formatted at 2026-08-06 16:16:42 on dist-test-slave-0b58"
server_key: "a516af905ed656df33d955b3709fd199"
server_key_iv: "e24c6be09a51753bac8c037f4c2dbbd6"
server_key_version: "encryptionkey@0"
I20260806 16:16:42.307425 9056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:42.328027 9056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:42.330999 9056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:42.331212 9056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:42.338471 9056 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.60:39825
I20260806 16:16:42.338769 9125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.60:39825 every 8 connection(s)
I20260806 16:16:42.339485 9056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
I20260806 16:16:42.342753 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:42.346313 8970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } attempt: 3
I20260806 16:16:42.346655 9126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:42.348562 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 9056
I20260806 16:16:42.348704 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032720994684-18792-0/minicluster-data/master-2/wal/instance
I20260806 16:16:42.349025 18792 external_mini_cluster.cc:1428] Setting key 8f3c85ba74fc7cf519f37f995ab5fbb3
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:16:42.360745 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.348433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.18.90.62:33735)
0806 16:16:42.348568 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.348575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.348984 (+ 409us) server_negotiation.cc:407] Connection header received
0806 16:16:42.349484 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.349490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.349543 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.349638 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.351000 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.351626 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353116 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.353451 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353545 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.356908 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.356911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.356913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.356959 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.358439 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.359157 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.359160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.359162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.359246 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.359570 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.359573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.359575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.359815 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.359925 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.360383 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.360534 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63,"spinlock_wait_cycles":896}
I20260806 16:16:42.361496 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.346862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33342)
0806 16:16:42.347039 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.347059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.347172 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.349623 (+ 2451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.349629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.349860 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.350011 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.350019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.352047 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.352050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.352717 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.352727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.353442 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.353582 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.354194 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.354216 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.355727 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.358582 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.358585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.358587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.358868 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.359306 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.359310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.359312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.359413 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.360169 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.360172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.360296 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.360406 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.361300 (+ 894us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":717,"spinlock_wait_cycles":5504}
I20260806 16:16:42.361744 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.347531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.18.90.60:39825)
0806 16:16:42.347938 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.347955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.347999 (+ 44us) server_negotiation.cc:407] Connection header received
0806 16:16:42.349127 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.349155 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.349417 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.350201 (+ 784us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.350701 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.351923 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.352846 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.353303 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353453 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.355888 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.355898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.355914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.355943 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.358446 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.359022 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.359029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.359031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.359164 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.359537 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.359542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.359547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.360052 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.360190 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.360537 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.360690 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":275,"thread_start_us":145,"threads_started":1}
I20260806 16:16:42.361745 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.348414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:33735 (local address 127.0.0.1:40372)
0806 16:16:42.348843 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.348900 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.349248 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.349827 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.349839 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.350310 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.350870 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.350882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.351759 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.351763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.352992 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.353003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.353707 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.353849 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.354472 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.354487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.356759 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.358554 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.358764 (+ 210us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.358773 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.359047 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.359340 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.359344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.359345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.359469 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.359924 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.359932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.360082 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.360303 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.360643 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":57,"spinlock_wait_cycles":27776,"thread_start_us":139,"threads_started":1}
I20260806 16:16:42.362040 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.349777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.18.90.60:39825)
0806 16:16:42.350072 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.350082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.350188 (+ 106us) server_negotiation.cc:407] Connection header received
0806 16:16:42.350306 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.350309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.350354 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.350447 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.351367 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.352048 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353173 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.353481 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.353579 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.357125 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.357129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.357130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.357151 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.359521 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.360005 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.360009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.360011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.360102 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.360437 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.360442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.360447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.360672 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.361593 (+ 921us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.361674 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.361796 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260806 16:16:42.363564 9126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:42.365124 9048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } attempt: 3
I20260806 16:16:42.365938 8970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81: Bootstrap starting.
I20260806 16:16:42.368494 8970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81: Neither blocks nor log segments found. Creating new log.
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:16:42.370326 8970 log.cc:824] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:42.373644 8970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81: No bootstrap required, opened a new log
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:42.374699 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.365530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33360)
0806 16:16:42.365640 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.365656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.365735 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.366049 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.366052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.366232 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.366321 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.366326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.367188 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.367192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.367789 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.367812 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.368423 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.368429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.368529 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.369075 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.369091 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.370639 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.372979 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.372983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.372985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.373218 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.373537 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.373540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.373542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.373632 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.374210 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.374213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.374303 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.374395 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.374527 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"spinlock_wait_cycles":3328}
I20260806 16:16:42.375324 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.365586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.18.90.60:39825)
0806 16:16:42.365856 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.365859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.365868 (+ 9us) server_negotiation.cc:407] Connection header received
0806 16:16:42.365894 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.365915 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.365944 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.366027 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.366436 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.367084 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.367975 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.368230 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.368404 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.370874 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.370881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.370887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.370919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.372857 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.373346 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.373349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.373351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.373432 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.373874 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.373877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.373878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.374094 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.374972 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.375044 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.375158 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:42.378017 8970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.378261 8970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e62684db19d4ae19856bc821c264c81, State: Initialized, Role: FOLLOWER
I20260806 16:16:42.378487 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.364170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.18.90.61:33537)
0806 16:16:42.364319 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.364322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.368918 (+ 4596us) server_negotiation.cc:407] Connection header received
0806 16:16:42.369053 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.369055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.369091 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.369170 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.369614 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.370360 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.371396 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.371753 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.371822 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.374306 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.374310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.374314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.374356 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.376425 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.376971 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.376977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.376980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.377067 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.377417 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.377421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.377424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.377711 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.377808 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.378187 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.378301 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":768}
I20260806 16:16:42.378531 9048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: Bootstrap starting.
I20260806 16:16:42.378592 8970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.379222 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.364054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:33537 (local address 127.0.0.1:57432)
0806 16:16:42.368836 (+ 4782us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.368849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.368946 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.369187 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.369190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.369361 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.369479 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.369486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.370487 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.370490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.371273 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.371282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.371966 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.371969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.372060 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.372531 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.372545 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.374151 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.376550 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.376554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.376557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.376848 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.377194 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.377198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.377200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.377301 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.377892 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.377897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.377995 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.378920 (+ 925us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.379063 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4699}
I20260806 16:16:42.380010 8970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:42.380038 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.369103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.18.90.62:33735)
0806 16:16:42.369241 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.369247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.369263 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:16:42.369301 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.369304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.369347 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.369428 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.370322 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.371075 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.372192 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.372594 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.372676 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.376156 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.376162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.376168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.376197 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.377991 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.378504 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.378508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.378510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.378598 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.378923 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.378927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.378930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.379140 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.379355 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.379754 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.379869 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:16:42.380155 9126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:42.382856 9048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:42.383780 9048 log.cc:824] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:42.384692 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.384825 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 16:16:42.386508 9144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:42.386588 9144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:42.388716 9048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: No bootstrap required, opened a new log
I20260806 16:16:42.385406 8901 master_runner.cc:186] Error getting master registration for 127.18.90.61:33537: OK, instance_id { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" instance_seqno: 1786033002109710 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:16:42.388964 8901 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 16:16:42 dist-test-slave-0b58 krb5kdc[8886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:16:42.392366 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.381210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.18.90.61:33537)
0806 16:16:42.381311 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.381315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.381390 (+ 75us) server_negotiation.cc:407] Connection header received
0806 16:16:42.381522 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.381525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.381571 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.381650 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.382463 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.383172 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.384374 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.384688 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.384762 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.388416 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.388420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.388423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.388449 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.390293 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.390774 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.390777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.390780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.390863 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.391306 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.391310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.391312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.391526 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.391661 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.392036 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.392155 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:16:42.392889 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.380651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:39825 (local address 127.0.0.1:33362)
0806 16:16:42.384347 (+ 3696us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:42.384363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:42.384457 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:42.384776 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:42.384780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:42.384962 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:42.385054 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.385061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.386075 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.386078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.386877 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:42.386887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.387522 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.387526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:42.387591 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.388060 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:42.388074 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:42.389607 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:42.391351 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.391354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.391356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.391542 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.391809 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:42.391813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:42.391815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:42.391893 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:42.392358 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:42.392360 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:42.392456 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 16:16:42.392560 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:16:42.392696 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3634}
I20260806 16:16:42.392889 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.380718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.18.90.60:39825)
0806 16:16:42.380834 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.380838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.384416 (+ 3578us) server_negotiation.cc:407] Connection header received
0806 16:16:42.384545 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.384548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.384592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.384666 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.385193 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.385952 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.387000 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.387327 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.387414 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.389730 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.389734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.389736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.389758 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.391215 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.391635 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.391640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.391642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.391721 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.391999 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.392002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.392003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.392200 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.392326 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.392603 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.392754 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":54}
I20260806 16:16:42.393903 9048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.394161 9048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff548bae76f44a5ba0289326f88232a, State: Initialized, Role: FOLLOWER
I20260806 16:16:42.394477 9048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.394963 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.395146 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:42.395758 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:16:42.395947 9048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:42.396042 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: Bootstrap starting.
I20260806 16:16:42.397104 8943 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8e62684db19d4ae19856bc821c264c81"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40386
I20260806 16:16:42.397358 8943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 16:16:42.397480 8943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:42.398826 9126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:42.398801 8943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.399737 9126 log.cc:824] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:42.399791 8943 leader_election.cc:288] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [CANDIDATE]: Term 1 election: Requested vote from peers 4ff548bae76f44a5ba0289326f88232a (127.18.90.61:33537), 5420cf263269416e82fa37b3b0d984c1 (127.18.90.60:39825)
I20260806 16:16:42.400756 9021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e62684db19d4ae19856bc821c264c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ff548bae76f44a5ba0289326f88232a"
I20260806 16:16:42.400977 9021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [term 0 FOLLOWER]: Advancing to term 1
W20260806 16:16:42.401228 9158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:42.401309 9158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:16:42.402963 9021 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e62684db19d4ae19856bc821c264c81 in term 1.
I20260806 16:16:42.404562 8917 leader_election.cc:302] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff548bae76f44a5ba0289326f88232a, 8e62684db19d4ae19856bc821c264c81; no voters:
I20260806 16:16:42.406373 9133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:42.406620 9133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [term 1 LEADER]: Becoming Leader. State: Replica: 8e62684db19d4ae19856bc821c264c81, State: Running, Role: LEADER
I20260806 16:16:42.407506 9133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.411870 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: No bootstrap required, opened a new log
I20260806 16:16:42.407923 8978 master_runner.cc:186] Error getting master registration for 127.18.90.60:39825: OK, instance_id { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" instance_seqno: 1786033002336494 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:16:42.411998 8978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:16:42.413872 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e62684db19d4ae19856bc821c264c81. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.414011 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:42.414414 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:42.415586 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:42.416227 9126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.416517 9126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5420cf263269416e82fa37b3b0d984c1, State: Initialized, Role: FOLLOWER
I20260806 16:16:42.416846 9126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } }
I20260806 16:16:42.418191 9126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:42.418709 9099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e62684db19d4ae19856bc821c264c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5420cf263269416e82fa37b3b0d984c1"
I20260806 16:16:42.418944 9099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:42.420881 9099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e62684db19d4ae19856bc821c264c81 in term 1.
I20260806 16:16:42.423025 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.407992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.18.90.62:33735)
0806 16:16:42.408103 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.408108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.409062 (+ 954us) server_negotiation.cc:407] Connection header received
0806 16:16:42.409351 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.409355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.409403 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.409453 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.410866 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.411601 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.413384 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.413712 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.413793 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.416655 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.416659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.416661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.416690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.418561 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.421598 (+ 3037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.421601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.421603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.421721 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.422042 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.422046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.422047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.422287 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.422416 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.422758 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.422879 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 16:16:42.423312 9099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [term 1 FOLLOWER]: Refusing update from remote peer 8e62684db19d4ae19856bc821c264c81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:42.423935 9133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:42.425446 9021 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [term 1 FOLLOWER]: Refusing update from remote peer 8e62684db19d4ae19856bc821c264c81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:42.425978 9133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:42.428869 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.411154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.18.90.60:39825)
0806 16:16:42.411361 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.411365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.414341 (+ 2976us) server_negotiation.cc:407] Connection header received
0806 16:16:42.414563 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.414567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.414622 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.414672 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.415988 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.416710 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.417965 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.418282 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.418358 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.425050 (+ 6692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.425055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.425058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.425084 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.426979 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.427516 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.427519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.427521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.427603 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.427899 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.427902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.427904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.428114 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.428229 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.428611 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.428722 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260806 16:16:42.429396 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e62684db19d4ae19856bc821c264c81. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.429527 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:42.430589 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.430707 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:42.432731 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.432838 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81 [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:42.433638 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.433770 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:42.434266 9172 mvcc.cc:205] Tried to move back new op lower bound from 7315591177918431232 to 7315591177883242496. Current Snapshot: MvccSnapshot[applied={T|T < 7315591177918431232}]
I20260806 16:16:42.434851 9171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.435019 9171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:42.435124 9171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e62684db19d4ae19856bc821c264c81. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.435207 9171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:42.435281 9171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8e62684db19d4ae19856bc821c264c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e62684db19d4ae19856bc821c264c81" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 33735 } } peers { permanent_uuid: "4ff548bae76f44a5ba0289326f88232a" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 33537 } } peers { permanent_uuid: "5420cf263269416e82fa37b3b0d984c1" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 39825 } } }
I20260806 16:16:42.435351 9171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:42.436802 9184 mvcc.cc:205] Tried to move back new op lower bound from 7315591177918431232 to 7315591177883242496. Current Snapshot: MvccSnapshot[applied={T|T < 7315591177918431232}]
I20260806 16:16:42.437266 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:42.404818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.18.90.61:33537)
0806 16:16:42.405759 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.405764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.408585 (+ 2821us) server_negotiation.cc:407] Connection header received
0806 16:16:42.409278 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.409283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.409333 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.409388 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.428345 (+ 18957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.429079 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.430974 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:42.431319 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:42.431395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:42.433260 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:42.433264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:42.433267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:42.433294 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:42.435091 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.435774 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.435777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.435779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.435860 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:42.436151 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:42.436155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:42.436156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:42.436385 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:42.436512 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:42.437000 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 16:16:42.437107 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850}
I20260806 16:16:42.438019 9170 catalog_manager.cc:1347] Generated new cluster ID: a88b8e98e23149aeae66749c23137e96
I20260806 16:16:42.438079 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260806 16:16:42.440611 9189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:42.440686 9189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:16:42.451399 9170 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:42.452723 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:42.454156 9186 mvcc.cc:205] Tried to move back new op lower bound from 7315591177918431232 to 7315591177883242496. Current Snapshot: MvccSnapshot[applied={T|T < 7315591177918431232}]
I20260806 16:16:42.462831 9170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e62684db19d4ae19856bc821c264c81: Generated new TSK 0
I20260806 16:16:42.463577 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:16:42.491003 8974 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.487990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.18.90.62:33735)
0806 16:16:42.488113 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.488118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.489931 (+ 1813us) server_negotiation.cc:407] Connection header received
0806 16:16:42.490127 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.490131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.490194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.490249 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.490868 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40396: BlockingRecv error: recv got EOF from 127.0.0.1:40396 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 16:16:42.492467 9128 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.488699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.18.90.60:39825)
0806 16:16:42.488805 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.488809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.491585 (+ 2776us) server_negotiation.cc:407] Connection header received
0806 16:16:42.491744 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.491749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.491794 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.491841 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.492354 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33370: BlockingRecv error: recv got EOF from 127.0.0.1:33370 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 16:16:42.493930 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.489338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.18.90.61:33537)
0806 16:16:42.489443 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.489447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.492831 (+ 3384us) server_negotiation.cc:407] Connection header received
0806 16:16:42.493215 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.493218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.493265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.493313 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.493812 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57456: BlockingRecv error: recv got EOF from 127.0.0.1:57456 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 16:16:42.498140 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.496744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.18.90.61:33537)
0806 16:16:42.496845 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.496849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.497273 (+ 424us) server_negotiation.cc:407] Connection header received
0806 16:16:42.497470 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.497473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.497542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.497590 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.498048 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57464: BlockingRecv error: recv got EOF from 127.0.0.1:57464 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 16:16:42.503118 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.501761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.18.90.61:33537)
0806 16:16:42.501856 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.501860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.502304 (+ 444us) server_negotiation.cc:407] Connection header received
0806 16:16:42.502470 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.502473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.502517 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.502564 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.503014 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57468: BlockingRecv error: recv got EOF from 127.0.0.1:57468 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 16:16:42.508771 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.507240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.18.90.61:33537)
0806 16:16:42.507347 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.507351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.507802 (+ 451us) server_negotiation.cc:407] Connection header received
0806 16:16:42.507976 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.507979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.508037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.508084 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.508672 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57476: BlockingRecv error: recv got EOF from 127.0.0.1:57476 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 16:16:42.516378 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.514831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.18.90.61:33537)
0806 16:16:42.514997 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.515002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.515480 (+ 478us) server_negotiation.cc:407] Connection header received
0806 16:16:42.515652 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.515656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.515700 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.515749 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.516220 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57480: BlockingRecv error: recv got EOF from 127.0.0.1:57480 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260806 16:16:42.528337 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.526689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.18.90.61:33537)
0806 16:16:42.526820 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.526824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.527344 (+ 520us) server_negotiation.cc:407] Connection header received
0806 16:16:42.527532 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.527535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.527584 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.527634 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.528166 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57486: BlockingRecv error: recv got EOF from 127.0.0.1:57486 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 16:16:42.548403 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.546718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.18.90.61:33537)
0806 16:16:42.546834 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.546839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.547359 (+ 520us) server_negotiation.cc:407] Connection header received
0806 16:16:42.547548 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.547552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.547604 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.547661 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.548247 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57488: BlockingRecv error: recv got EOF from 127.0.0.1:57488 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 16:16:42.584945 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.583114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.18.90.61:33537)
0806 16:16:42.583260 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.583265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.583869 (+ 604us) server_negotiation.cc:407] Connection header received
0806 16:16:42.584090 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.584094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.584150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.584219 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.584823 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57490: BlockingRecv error: recv got EOF from 127.0.0.1:57490 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 16:16:42.652568 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.651431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.18.90.61:33537)
0806 16:16:42.651625 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.651629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.651713 (+ 84us) server_negotiation.cc:407] Connection header received
0806 16:16:42.651834 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.651836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.651884 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.651954 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.652425 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57500: BlockingRecv error: recv got EOF from 127.0.0.1:57500 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260806 16:16:42.783812 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:42.782728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.18.90.61:33537)
0806 16:16:42.782869 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:42.782873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:42.783050 (+ 177us) server_negotiation.cc:407] Connection header received
0806 16:16:42.783160 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:42.783163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:42.783219 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:42.783301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:42.783713 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57508: BlockingRecv error: recv got EOF from 127.0.0.1:57508 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 16:16:43.043541 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:43.042313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.18.90.61:33537)
0806 16:16:43.042462 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:43.042467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:43.042598 (+ 131us) server_negotiation.cc:407] Connection header received
0806 16:16:43.042760 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:43.042763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:43.042818 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:43.042903 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:43.043388 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57512: BlockingRecv error: recv got EOF from 127.0.0.1:57512 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:16:43.402602 9158 catalog_manager.cc:1259] Loaded cluster ID: a88b8e98e23149aeae66749c23137e96
I20260806 16:16:43.402720 9158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: loading cluster ID for follower catalog manager: success
I20260806 16:16:43.404371 9158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: acquiring CA information for follower catalog manager: success
I20260806 16:16:43.405483 9158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ff548bae76f44a5ba0289326f88232a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:16:43.442018 9189 catalog_manager.cc:1259] Loaded cluster ID: a88b8e98e23149aeae66749c23137e96
I20260806 16:16:43.442098 9189 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: loading cluster ID for follower catalog manager: success
I20260806 16:16:43.443815 9189 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: acquiring CA information for follower catalog manager: success
I20260806 16:16:43.445000 9189 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5420cf263269416e82fa37b3b0d984c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:16:43.561200 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:43.557641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.18.90.61:33537)
0806 16:16:43.557903 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:43.557907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:43.557921 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:16:43.558042 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:43.558044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:43.558171 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:43.558276 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:43.558504 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.559020 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.559808 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.560022 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.560097 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:43.560440 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:43.560885 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:43.560988 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:43.561041 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 16:16:43.561084 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260806 16:16:43.567421 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:43.564046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.18.90.60:39825)
0806 16:16:43.564317 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:43.564321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:43.564333 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:16:43.564407 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:43.564410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:43.564511 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:43.564596 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:43.564764 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.565365 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.566104 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.566344 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.566418 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:43.566666 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:43.567039 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:43.567125 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:43.567240 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 16:16:43.567308 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260806 16:16:43.573933 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:43.570151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.18.90.61:33537)
0806 16:16:43.570280 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:43.570284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:43.570621 (+ 337us) server_negotiation.cc:407] Connection header received
0806 16:16:43.570670 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:43.570673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:43.570720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:43.570787 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:43.571111 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.571878 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.572850 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.573144 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.573246 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:43.573474 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:43.573559 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:43.573623 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:43.573752 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 16:16:43.573813 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:16:43.574433 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:43.570469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.18.90.60:39825)
0806 16:16:43.570585 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:43.570590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:43.570910 (+ 320us) server_negotiation.cc:407] Connection header received
0806 16:16:43.571050 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:43.571054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:43.571101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:43.571162 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:43.571422 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.572092 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.573152 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.573493 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.573584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:43.573884 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:43.573959 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:43.574020 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:43.574261 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 16:16:43.574308 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 16:16:43.574889 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:43.570126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.18.90.62:33735)
0806 16:16:43.570372 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:43.570376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:43.570453 (+ 77us) server_negotiation.cc:407] Connection header received
0806 16:16:43.570662 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:43.570665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:43.570827 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:43.570948 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:43.571288 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.571867 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.573043 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:43.573364 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:43.573470 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:43.573772 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:43.574085 (+ 313us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:43.574179 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:43.574734 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 16:16:43.574785 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":66,"threads_started":1}
I20260806 16:16:43.581415 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8901
I20260806 16:16:43.589350 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 8978
I20260806 16:16:43.596001 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 9056
2026-08-06T16:16:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3059 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:16:43 dist-test-slave-0b58 krb5kdc[9280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:16:43 dist-test-slave-0b58 krb5kdc[9280](info): set up 2 sockets
Aug 06 16:16:43 dist-test-slave-0b58 krb5kdc[9280](info): commencing operation
krb5kdc: starting...
W20260806 16:16:44.632737 18792 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/test-user.keytab.
Aug 06 16:16:44 dist-test-slave-0b58 krb5kdc[9280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:16:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.90.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
Entry for principal kudu/127.18.90.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab.
I20260806 16:16:44.758481 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42513
--webserver_interface=127.18.90.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42513,127.18.90.61:45363,127.18.90.60:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:44.871399 9293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:16:44.871683 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:44.871706 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:44.871744 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:16:44.875710 9293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:44.875772 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:44.875792 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:44.875808 9293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:44.875823 9293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:44.881157 9293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42513,127.18.90.61:45363,127.18.90.60:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.62.keytab
--principal=kudu/127.18.90.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.62:42513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.90.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:44.882220 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:44.883190 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:44.891897 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:44.891897 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:44.891934 9293 server_base.cc:1034] running on GCE node
W20260806 16:16:44.891912 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:44.892848 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:44.893975 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:44.895174 9293 hybrid_clock.cc:648] HybridClock initialized: now 1786033004895154 us; error 32 us; skew 500 ppm
Aug 06 16:16:44 dist-test-slave-0b58 krb5kdc[9280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:44.898008 9293 init.cc:377] Logged in from keytab as kudu/127.18.90.62@KRBTEST.COM (short username kudu)
I20260806 16:16:44.898977 9293 webserver.cc:460] Webserver started at http://127.18.90.62:42785/ using document root <none> and password file <none>
I20260806 16:16:44.899561 9293 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:44.899655 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:44.899899 9293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:44.902161 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data/instance:
uuid: "84a791db69084487a71b65f282a4c83e"
format_stamp: "Formatted at 2026-08-06 16:16:44 on dist-test-slave-0b58"
server_key: "5d48c675d7654fda7da6d4f315c840bc"
server_key_iv: "af9c1027e1810ad2e9561739136eff00"
server_key_version: "encryptionkey@0"
I20260806 16:16:44.902765 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal/instance:
uuid: "84a791db69084487a71b65f282a4c83e"
format_stamp: "Formatted at 2026-08-06 16:16:44 on dist-test-slave-0b58"
server_key: "5d48c675d7654fda7da6d4f315c840bc"
server_key_iv: "af9c1027e1810ad2e9561739136eff00"
server_key_version: "encryptionkey@0"
I20260806 16:16:44.906720 9293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:16:44.909140 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:44.910344 9293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:44.910495 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal
uuid: "84a791db69084487a71b65f282a4c83e"
format_stamp: "Formatted at 2026-08-06 16:16:44 on dist-test-slave-0b58"
server_key: "5d48c675d7654fda7da6d4f315c840bc"
server_key_iv: "af9c1027e1810ad2e9561739136eff00"
server_key_version: "encryptionkey@0"
I20260806 16:16:44.910609 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:44.923041 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:44.926039 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:44.926301 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:44.933874 9293 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.62:42513
I20260806 16:16:44.934185 9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.62:42513 every 8 connection(s)
I20260806 16:16:44.934908 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/data/info.pb
I20260806 16:16:44.935245 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 9293
I20260806 16:16:44.935372 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-0/wal/instance
I20260806 16:16:44.935675 18792 external_mini_cluster.cc:1428] Setting key 7762ec5ffd4f65f0578cfed93fe26a96
I20260806 16:16:44.937831 9364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:44.943449 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.90.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.61@KRBTEST.COM" created.
Aug 06 16:16:44 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:44.960704 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:44.946368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42513 (local address 127.0.0.1:44704)
0806 16:16:44.946922 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:44.946962 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:44.947758 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:44.948539 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:44.948544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:44.948987 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:44.949246 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:44.949259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:44.950745 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:44.950748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:44.951483 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:44.951493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:44.952177 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:44.952181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:44.952382 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:44.953037 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:44.953053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:44.954948 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:44.957016 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:44.957026 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:44.957038 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:44.957382 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:44.957739 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:44.957744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:44.957748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:44.957921 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:44.958724 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:44.958733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:44.958911 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 16:16:44.959103 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 16:16:44.959607 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":134,"spinlock_wait_cycles":15232,"thread_start_us":204,"threads_started":1}
I20260806 16:16:44.960709 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:44.946368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.18.90.62:42513)
0806 16:16:44.946875 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:44.946893 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:44.947071 (+ 178us) server_negotiation.cc:407] Connection header received
0806 16:16:44.948052 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:44.948056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:44.948350 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:44.948539 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:44.949412 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:44.950605 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:44.951596 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:44.951993 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:44.952192 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:44.955112 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:44.955118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:44.955128 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:44.955149 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:44.956895 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:44.957502 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:44.957507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:44.957508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:44.957608 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:44.958030 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:44.958032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:44.958033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:44.958591 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:44.958731 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:44.959205 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 16:16:44.959483 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":358,"thread_start_us":129,"threads_started":1}
I20260806 16:16:44.969451 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:44.970404 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:44.970037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39714)
0806 16:16:44.970188 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:44.970295 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:16:44.970649 9312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
I20260806 16:16:44.971593 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:44.971384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39716)
0806 16:16:44.971483 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:44.971531 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 16:16:44.971730 9364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:45363: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab.
Entry for principal kudu/127.18.90.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab.
I20260806 16:16:44.973162 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:45363
--webserver_interface=127.18.90.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab
--principal=kudu/127.18.90.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42513,127.18.90.61:45363,127.18.90.60:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:16:45.021065 9364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } attempt: 1
I20260806 16:16:45.022087 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.021700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39732)
0806 16:16:45.021925 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.021995 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 16:16:45.025018 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.024731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39742)
0806 16:16:45.024891 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.024940 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260806 16:16:45.025223 9364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:45363: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
W20260806 16:16:45.090370 9371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:16:45.090706 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:45.090791 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:45.090883 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:16:45.093413 9364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } attempt: 2
I20260806 16:16:45.094280 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.093975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39758)
0806 16:16:45.094133 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.094202 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 16:16:45.095002 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.094783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39762)
0806 16:16:45.094892 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.094944 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:16:45.095144 9364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.61:45363: Network error: Client connection negotiation failed: client connection to 127.18.90.61:45363: connect: Connection refused (error 111)
W20260806 16:16:45.095346 9371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:45.095391 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:45.095428 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:45.095461 9371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:45.095490 9371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:45.101226 9371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42513,127.18.90.61:45363,127.18.90.60:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.61.keytab
--principal=kudu/127.18.90.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.61:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.90.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:45.102357 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:45.103399 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:45.112303 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:45.112255 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:45.112283 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:45.112538 9371 server_base.cc:1034] running on GCE node
I20260806 16:16:45.113060 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:45.114168 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:45.115396 9371 hybrid_clock.cc:648] HybridClock initialized: now 1786033005115365 us; error 49 us; skew 500 ppm
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:45.118297 9371 init.cc:377] Logged in from keytab as kudu/127.18.90.61@KRBTEST.COM (short username kudu)
I20260806 16:16:45.119168 9371 webserver.cc:460] Webserver started at http://127.18.90.61:35795/ using document root <none> and password file <none>
I20260806 16:16:45.119735 9371 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:45.119851 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:45.120090 9371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:45.122229 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data/instance:
uuid: "5230806d085e444f81ba92040971a3dc"
format_stamp: "Formatted at 2026-08-06 16:16:45 on dist-test-slave-0b58"
server_key: "feb92ed82ef2b4ee85c28870e0dcf1bc"
server_key_iv: "6bf892639c2875a95637f69d316056bd"
server_key_version: "encryptionkey@0"
I20260806 16:16:45.122874 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal/instance:
uuid: "5230806d085e444f81ba92040971a3dc"
format_stamp: "Formatted at 2026-08-06 16:16:45 on dist-test-slave-0b58"
server_key: "feb92ed82ef2b4ee85c28870e0dcf1bc"
server_key_iv: "6bf892639c2875a95637f69d316056bd"
server_key_version: "encryptionkey@0"
I20260806 16:16:45.126700 9371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 16:16:45.129153 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:45.130381 9371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:16:45.130537 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal
uuid: "5230806d085e444f81ba92040971a3dc"
format_stamp: "Formatted at 2026-08-06 16:16:45 on dist-test-slave-0b58"
server_key: "feb92ed82ef2b4ee85c28870e0dcf1bc"
server_key_iv: "6bf892639c2875a95637f69d316056bd"
server_key_version: "encryptionkey@0"
I20260806 16:16:45.130674 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:45.142098 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:45.145020 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:45.145226 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:45.152215 9371 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.61:45363
I20260806 16:16:45.152546 9440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.61:45363 every 8 connection(s)
I20260806 16:16:45.153216 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/data/info.pb
I20260806 16:16:45.155905 9441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:45.159643 9441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:45.160372 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 9371
I20260806 16:16:45.160502 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-1/wal/instance
I20260806 16:16:45.160802 18792 external_mini_cluster.cc:1428] Setting key d49304f204d89ec4afe8a25acaf6db96
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.90.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.90.60@KRBTEST.COM" created.
I20260806 16:16:45.175532 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.161960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.18.90.62:42513)
0806 16:16:45.162145 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.162152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.162958 (+ 806us) server_negotiation.cc:407] Connection header received
0806 16:16:45.164011 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.164014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.164082 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.164193 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.165835 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.166470 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.167822 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.168066 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.168184 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.170745 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.170751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.170756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.170794 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.172536 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.173089 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.173093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.173095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.173213 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.173612 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.173615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.173617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.173820 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.173902 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.175157 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.175304 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:16:45.176077 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.162327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42513 (local address 127.0.0.1:44708)
0806 16:16:45.162805 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.162858 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.163790 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.164402 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.164413 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.164952 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.165658 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.165680 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.166625 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.166632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.167682 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.167693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.168229 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.168233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.168440 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.169086 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.169102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.170526 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.172677 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.172683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.172699 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.172975 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.173335 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.173340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.173343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.173508 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.173937 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.173946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.174309 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.174537 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.174912 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260806 16:16:45.177695 9441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:45.186354 9364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } attempt: 3
I20260806 16:16:45.189407 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.178266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39772)
0806 16:16:45.178465 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.178481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.178563 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.179097 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.179100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.179279 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.179361 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.179366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.180385 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.180390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.180925 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.180933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.181539 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.181543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.181635 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.182141 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.182152 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.185094 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.187392 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.187395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.187397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.187653 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.188358 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.188361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.188363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.188457 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.189006 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.189008 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.189096 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.189180 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.189284 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 16:16:45.189906 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.178388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.18.90.61:45363)
0806 16:16:45.178766 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.178778 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.178796 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:16:45.178829 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.178834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.178980 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.179122 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.179468 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.180193 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.181060 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.181428 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.181539 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.185259 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.185268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.185281 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.185304 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.187284 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.187783 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.187789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.187791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.187931 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.188567 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.188570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.188571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.188894 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.189035 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.189309 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.189467 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":176,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.90.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab.
Entry for principal kudu/127.18.90.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab.
I20260806 16:16:45.191948 9441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:45.192720 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.192439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54448)
0806 16:16:45.192547 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.192624 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 16:16:45.192853 18792 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG62Umc/build/debug/bin/kudu
/tmp/dist-test-taskG62Umc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:37513
--webserver_interface=127.18.90.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab
--principal=kudu/127.18.90.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.90.20:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.90.62:42513,127.18.90.61:45363,127.18.90.60:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:16:45.192909 9389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
I20260806 16:16:45.193886 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.193651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54450)
0806 16:16:45.193745 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.193797 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 16:16:45.194025 9441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:37513: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
I20260806 16:16:45.197124 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.186986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.18.90.61:45363)
0806 16:16:45.187316 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.187322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.187332 (+ 10us) server_negotiation.cc:407] Connection header received
0806 16:16:45.187385 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.187388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.187431 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.187534 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.188041 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.188801 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.189936 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.190344 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.190429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.193379 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.193386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.193388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.193407 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.195402 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.195840 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.195846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.195848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.195946 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.196245 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.196248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.196249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.196493 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.196573 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.196916 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.197011 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260806 16:16:45.197278 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.186897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39776)
0806 16:16:45.187011 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.187026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.187109 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.187577 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.187579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.187752 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.187906 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.187916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.188907 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.188911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.189806 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.189819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.190458 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.190462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.190613 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.191089 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.191106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.192614 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.195522 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.195524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.195526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.195739 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.196042 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.196044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.196046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.196125 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.196595 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.196601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.196745 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.196866 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.197038 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 16:16:45.198123 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:45.198949 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.198624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54454)
0806 16:16:45.198785 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.198857 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 16:16:45.199733 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.199399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54470)
0806 16:16:45.199574 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.199639 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260806 16:16:45.199898 9364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:37513: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
I20260806 16:16:45.231093 9364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } attempt: 1
I20260806 16:16:45.232033 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.231666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54476)
0806 16:16:45.231853 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.231935 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 16:16:45.232861 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.232607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54478)
0806 16:16:45.232729 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.232785 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 16:16:45.233018 9364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:37513: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
I20260806 16:16:45.243381 9441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } attempt: 1
I20260806 16:16:45.244344 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.243994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54484)
0806 16:16:45.244150 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.244211 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 16:16:45.245092 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.244856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54492)
0806 16:16:45.244969 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.245021 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 16:16:45.245204 9441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:37513: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
I20260806 16:16:45.308223 9364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } attempt: 2
I20260806 16:16:45.309155 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.308849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54504)
0806 16:16:45.309011 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.309077 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 16:16:45.309918 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.309677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54514)
0806 16:16:45.309810 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.309858 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 16:16:45.310148 9364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:37513: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
W20260806 16:16:45.310858 9449 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:16:45.311156 9449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:16:45.311209 9449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:16:45.311291 9449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:16:45.313356 9441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } attempt: 2
I20260806 16:16:45.314034 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.313790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54522)
0806 16:16:45.313912 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.313970 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 16:16:45.314793 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.314586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54536)
0806 16:16:45.314701 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.314744 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 16:16:45.314934 9441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.90.60:37513: Network error: Client connection negotiation failed: client connection to 127.18.90.60:37513: connect: Connection refused (error 111)
W20260806 16:16:45.315869 9449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:16:45.315950 9449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:16:45.316015 9449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:16:45.316079 9449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:16:45.316118 9449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:16:45.321714 9449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.90.20:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.90.62:42513,127.18.90.61:45363,127.18.90.60:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/krb5kdc/kudu_127.18.90.60.keytab
--principal=kudu/127.18.90.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.90.60:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.90.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:19:10 UTC on ccycloud.quasar-onbjwb.root.comops.site
build id 4570623
I20260806 16:16:45.322861 9449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:16:45.323940 9449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:16:45.332682 9454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:45.332717 9455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:16:45.332671 9457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:16:45.333060 9449 server_base.cc:1034] running on GCE node
I20260806 16:16:45.333591 9449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:16:45.334843 9449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:16:45.336046 9449 hybrid_clock.cc:648] HybridClock initialized: now 1786033005336027 us; error 38 us; skew 500 ppm
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:16:45.338955 9449 init.cc:377] Logged in from keytab as kudu/127.18.90.60@KRBTEST.COM (short username kudu)
I20260806 16:16:45.339880 9449 webserver.cc:460] Webserver started at http://127.18.90.60:44281/ using document root <none> and password file <none>
I20260806 16:16:45.340490 9449 fs_manager.cc:359] Metadata directory not provided
I20260806 16:16:45.340557 9449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:16:45.340729 9449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:16:45.342734 9449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data/instance:
uuid: "27eb45ec2df94e2a9e94fef185340b59"
format_stamp: "Formatted at 2026-08-06 16:16:45 on dist-test-slave-0b58"
server_key: "e938bd1510bd4232cea76c52c1009229"
server_key_iv: "97e30c778acc6e33b407cd52d30fab79"
server_key_version: "encryptionkey@0"
I20260806 16:16:45.343309 9449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal/instance:
uuid: "27eb45ec2df94e2a9e94fef185340b59"
format_stamp: "Formatted at 2026-08-06 16:16:45 on dist-test-slave-0b58"
server_key: "e938bd1510bd4232cea76c52c1009229"
server_key_iv: "97e30c778acc6e33b407cd52d30fab79"
server_key_version: "encryptionkey@0"
I20260806 16:16:45.347115 9449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:16:45.349555 9464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:16:45.350788 9449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:16:45.350893 9449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data,/tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal
uuid: "27eb45ec2df94e2a9e94fef185340b59"
format_stamp: "Formatted at 2026-08-06 16:16:45 on dist-test-slave-0b58"
server_key: "e938bd1510bd4232cea76c52c1009229"
server_key_iv: "97e30c778acc6e33b407cd52d30fab79"
server_key_version: "encryptionkey@0"
I20260806 16:16:45.350982 9449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:16:45.387981 9449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:16:45.391749 9449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:16:45.391913 9449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:16:45.399247 9449 rpc_server.cc:307] RPC server started. Bound to: 127.18.90.60:37513
I20260806 16:16:45.399526 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.90.60:37513 every 8 connection(s)
I20260806 16:16:45.400362 9449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/data/info.pb
I20260806 16:16:45.401556 18792 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG62Umc/build/debug/bin/kudu as pid 9449
I20260806 16:16:45.401693 18792 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG62Umc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032720994684-18792-0/minicluster-data/master-2/wal/instance
I20260806 16:16:45.402037 18792 external_mini_cluster.cc:1428] Setting key c312973f3a976818e48d4678eb2ab803
I20260806 16:16:45.403784 9519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:16:45.406107 9441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } attempt: 3
I20260806 16:16:45.408629 9519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:45.409368 9364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } attempt: 3
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.61@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.62@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:45.419407 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.406611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54552)
0806 16:16:45.406743 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.406760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.406843 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.407224 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.407227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.407424 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.407541 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.407548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.408523 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.408526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.409206 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.409217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.410410 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.410414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.410495 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.411120 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.411136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.412925 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.416985 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.416989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.416991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.417272 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.417670 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.417676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.417679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.417778 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.418769 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.418775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.418896 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.419019 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.419181 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 16:16:45.421222 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.403405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.18.90.60:37513)
0806 16:16:45.403814 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.403852 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.403906 (+ 54us) server_negotiation.cc:407] Connection header received
0806 16:16:45.404651 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.404670 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.405013 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.405336 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.406311 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.408083 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.409371 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.410216 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.410380 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.413737 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.413740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.413743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.413770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.416739 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.418005 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.418008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.418010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.418094 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.418437 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.418440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.418442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.418643 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.418811 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.419627 (+ 816us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.420039 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
I20260806 16:16:45.421242 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.406670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.18.90.60:37513)
0806 16:16:45.406998 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.407006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.407019 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:16:45.407052 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.407054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.407092 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.407183 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.407665 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.408419 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.409362 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.410260 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.410736 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.413129 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.413136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.413149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.413181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.416859 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.417390 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.417396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.417402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.417558 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.417898 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.417906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.417909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.418569 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.418711 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.420106 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.420240 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":339,"server-negotiator.queue_time_us":246,"thread_start_us":159,"threads_started":1}
I20260806 16:16:45.422027 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.409837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54568)
0806 16:16:45.409977 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.409995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.410078 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.412046 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.412049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.412248 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.412404 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.412413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.413650 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.413655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.414596 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.414607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.415331 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.415337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.415436 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.415896 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.415913 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.417454 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.420197 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.420200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.420203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.420629 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.420944 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.420947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.420949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.421038 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.421496 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.421499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.421616 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.421777 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.421911 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 16:16:45.422516 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.411025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.18.90.62:42513)
0806 16:16:45.411137 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.411141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.411468 (+ 327us) server_negotiation.cc:407] Connection header received
0806 16:16:45.411655 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.411658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.411712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.411787 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.412882 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.413906 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.415070 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.415961 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.416087 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.418793 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.418796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.418799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.418827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.420615 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.421174 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.421177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.421179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.421284 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.421599 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.421603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.421604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.421827 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.421899 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.422298 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.422413 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:16:45.423032 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.409884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.18.90.60:37513)
0806 16:16:45.411026 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.411036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.411050 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:16:45.411821 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.411825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.411879 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.412548 (+ 669us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.412767 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.413543 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.414727 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.415125 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.415640 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.417613 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.417620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.417622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.417648 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.420074 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.420754 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.420757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.420758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.420841 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.421147 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.421149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.421151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.421383 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.422529 (+ 1146us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.422668 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.422789 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":1038,"thread_start_us":92,"threads_started":1}
I20260806 16:16:45.423753 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.411011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.62:42513 (local address 127.0.0.1:44724)
0806 16:16:45.411353 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.411400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.411548 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.411985 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.411991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.412467 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.412747 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.412757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.414015 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.414020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.414954 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.414965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.416183 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.416187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.416351 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.416937 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.416952 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.418622 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.420735 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.420743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.420748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.421055 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.421401 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.421404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.421405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.421496 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.421974 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.421979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.422082 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.423403 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.423583 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":56,"spinlock_wait_cycles":7936,"thread_start_us":79,"threads_started":1}
I20260806 16:16:45.425745 9364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e: Bootstrap starting.
I20260806 16:16:45.425743 9441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: Bootstrap starting.
I20260806 16:16:45.427894 9441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:45.427896 9364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:45.428683 9441 log.cc:824] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:45.428700 9364 log.cc:824] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:45.429502 9519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:45.430907 9364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e: No bootstrap required, opened a new log
I20260806 16:16:45.430907 9441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: No bootstrap required, opened a new log
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.62@KRBTEST.COM
I20260806 16:16:45.435076 9441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.435334 9441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5230806d085e444f81ba92040971a3dc, State: Initialized, Role: FOLLOWER
I20260806 16:16:45.435077 9364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.435652 9364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a791db69084487a71b65f282a4c83e, State: Initialized, Role: FOLLOWER
I20260806 16:16:45.435688 9441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.436107 9364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:45.437165 9441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:45.437506 9364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:45.439410 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.428490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.18.90.62:42513)
0806 16:16:45.428624 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.428629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.428644 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:16:45.428681 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.428683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.428734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.428819 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.429749 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.430475 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.431665 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.431983 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.432058 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.435146 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.435150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.435154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.435184 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.436917 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.437440 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.437445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.437448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.437542 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.437871 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.437875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.437878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.438132 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.438341 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.439083 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.439214 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:16:45.439647 9529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.440076 9529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:45.441234 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.430307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.18.90.61:45363)
0806 16:16:45.430410 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.430414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.430602 (+ 188us) server_negotiation.cc:407] Connection header received
0806 16:16:45.430905 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.430909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.430993 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.431046 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.431791 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.432897 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.434093 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.434487 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.434573 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.437566 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.437573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.437579 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.437608 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.439223 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.439730 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.439733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.439735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.439844 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.440242 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.440246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.440248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.440517 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.440608 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.440956 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.441086 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:16:45.441665 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.430003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.61:45363 (local address 127.0.0.1:39790)
0806 16:16:45.430494 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.430553 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.430733 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.431230 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.431322 (+ 92us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.431524 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.431659 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.431667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.433029 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.433035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.433955 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.433967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.434783 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.434790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.434899 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.435384 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.435402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.437422 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.439365 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.439368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.439370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.439602 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.439997 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.440002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.440005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.440122 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.440656 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.440659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.440760 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.441357 (+ 597us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.441503 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425}
W20260806 16:16:45.443658 9552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:45.443658 9551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:45.443737 9551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 16:16:45.443887 9552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:16:45.443907 9519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:16:45.444203 9530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.444393 9530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: This master's current role is: FOLLOWER
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.90.61@KRBTEST.COM
I20260806 16:16:45.452553 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.442201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.18.90.61:45363)
0806 16:16:45.442332 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.442338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.442354 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:16:45.442393 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.442398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.442441 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.442512 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.443382 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.444009 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.445040 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.445353 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.445450 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.448505 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.448509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.448511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.448537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.450331 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.450829 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.450833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.450834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.450919 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.451243 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.451246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.451248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.451474 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.451646 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.452250 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.452400 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:16:45.451570 9293 master_runner.cc:186] Error getting master registration for 127.18.90.60:37513: OK, instance_id { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" instance_seqno: 1786033005397129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:16:45.456723 9293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:16:45.455802 9371 master_runner.cc:186] Error getting master registration for 127.18.90.60:37513: OK, instance_id { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" instance_seqno: 1786033005397129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:16:45.459968 9371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:16:45.461833 18792 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 06 16:16:45 dist-test-slave-0b58 krb5kdc[9280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786033005, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.90.60@KRBTEST.COM for kudu/127.18.90.60@KRBTEST.COM
I20260806 16:16:45.463471 9337 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "84a791db69084487a71b65f282a4c83e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44728
I20260806 16:16:45.463743 9337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 16:16:45.463868 9337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:45.465251 9337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.466235 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.444398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.18.90.60:37513)
0806 16:16:45.444502 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.444506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.457019 (+ 12513us) server_negotiation.cc:407] Connection header received
0806 16:16:45.457164 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.457170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.457220 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.457266 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.457768 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.458542 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.459535 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.459923 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.460008 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.462706 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.462710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.462713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.462739 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.464578 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.465041 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.465044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.465046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.465111 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.465400 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.465403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.465404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.465601 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.465669 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.465935 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.466050 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:16:45.466259 9337 leader_election.cc:288] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [CANDIDATE]: Term 1 election: Requested vote from peers 5230806d085e444f81ba92040971a3dc (127.18.90.61:45363), 27eb45ec2df94e2a9e94fef185340b59 (127.18.90.60:37513)
I20260806 16:16:45.466707 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.444578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.90.60:37513 (local address 127.0.0.1:54584)
0806 16:16:45.456952 (+ 12374us) negotiation.cc:107] Waiting for socket to connect
0806 16:16:45.456968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:16:45.457078 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:16:45.457380 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:16:45.457383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:16:45.457566 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:16:45.457649 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.457656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.458659 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.458663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.459411 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:16:45.459423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.460320 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.460323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:16:45.460403 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.460938 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:16:45.460953 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:16:45.462570 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:16:45.464686 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.464689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.464691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.464948 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.465230 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:16:45.465232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:16:45.465234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:16:45.465311 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:16:45.465724 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:16:45.465727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:16:45.465796 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 16:16:45.466432 (+ 636us) client_negotiation.cc:240] Negotiation successful
0806 16:16:45.466562 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12289}
I20260806 16:16:45.467293 9415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84a791db69084487a71b65f282a4c83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5230806d085e444f81ba92040971a3dc"
I20260806 16:16:45.467536 9415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:45.470719 9415 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a791db69084487a71b65f282a4c83e in term 1.
I20260806 16:16:45.471140 9312 leader_election.cc:302] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5230806d085e444f81ba92040971a3dc, 84a791db69084487a71b65f282a4c83e; no voters:
I20260806 16:16:45.471463 9530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:16:45.471707 9530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [term 1 LEADER]: Becoming Leader. State: Replica: 84a791db69084487a71b65f282a4c83e, State: Running, Role: LEADER
I20260806 16:16:45.472064 9530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.472746 9519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: Bootstrap starting.
I20260806 16:16:45.473973 9530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a791db69084487a71b65f282a4c83e. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.474085 9530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:45.475564 9519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: Neither blocks nor log segments found. Creating new log.
I20260806 16:16:45.476521 9519 log.cc:824] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: Log is configured to *not* fsync() on all Append() calls
I20260806 16:16:45.478061 9564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:16:45.478996 9564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:16:45.485101 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.471720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.18.90.62:42513)
0806 16:16:45.471846 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.471850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.473685 (+ 1835us) server_negotiation.cc:407] Connection header received
0806 16:16:45.473886 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.473889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.473937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.473987 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.474991 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.475717 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.477640 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.477978 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.478053 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.480821 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.480825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.480828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.480854 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.482542 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.483260 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.483265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.483267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.483361 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.483824 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.483828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.483830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.484041 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.484167 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.484818 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.484950 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:16:45.490173 9519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: No bootstrap required, opened a new log
I20260806 16:16:45.492308 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.472442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.18.90.61:45363)
0806 16:16:45.472549 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.472553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.475262 (+ 2709us) server_negotiation.cc:407] Connection header received
0806 16:16:45.476859 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.476863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.476913 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.476964 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.480374 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.481219 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.482366 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.482689 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.482757 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.488112 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.488116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.488118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.488153 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.489847 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.490589 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.490593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.490595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.490682 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.491116 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.491120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.491122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.491353 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.491466 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.491970 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.492101 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:16:45.495792 9519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.496068 9519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27eb45ec2df94e2a9e94fef185340b59, State: Initialized, Role: FOLLOWER
I20260806 16:16:45.496474 9519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } }
I20260806 16:16:45.497097 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.497381 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:45.497985 9519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:16:45.498049 9491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84a791db69084487a71b65f282a4c83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27eb45ec2df94e2a9e94fef185340b59"
I20260806 16:16:45.498301 9491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:16:45.499722 9491 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a791db69084487a71b65f282a4c83e in term 1.
I20260806 16:16:45.505537 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:45.473083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.18.90.60:37513)
0806 16:16:45.473183 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.473188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.483394 (+ 10206us) server_negotiation.cc:407] Connection header received
0806 16:16:45.483582 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.483586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.483635 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.483683 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.485420 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.486217 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.487748 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:45.488118 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:45.488238 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:45.501582 (+ 13344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:16:45.501587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:16:45.501590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:16:45.501621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:16:45.503464 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.503994 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.503998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.504000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.504094 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:16:45.504474 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:16:45.504478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:16:45.504479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:16:45.504729 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:16:45.504849 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:45.505250 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 16:16:45.505385 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 16:16:45.508018 9491 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [term 1 FOLLOWER]: Refusing update from remote peer 84a791db69084487a71b65f282a4c83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:45.508057 9415 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [term 1 FOLLOWER]: Refusing update from remote peer 84a791db69084487a71b65f282a4c83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:16:45.508701 9559 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:16:45.509351 9530 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 16:16:45.510576 9578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:16:45.510653 9578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:16:45.514020 9529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a791db69084487a71b65f282a4c83e. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.514125 9529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:45.518137 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a791db69084487a71b65f282a4c83e. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.518292 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:45.525151 9530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.525314 9530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:45.525456 9566 mvcc.cc:205] Tried to move back new op lower bound from 7315591190552231936 to 7315591190417383424. Current Snapshot: MvccSnapshot[applied={T|T < 7315591190552231936}]
I20260806 16:16:45.526809 9529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.526942 9529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:45.529489 9580 mvcc.cc:205] Tried to move back new op lower bound from 7315591190552231936 to 7315591190417383424. Current Snapshot: MvccSnapshot[applied={T|T < 7315591190552231936}]
I20260806 16:16:45.533499 9559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.533636 9559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e [sys.catalog]: This master's current role is: LEADER
I20260806 16:16:45.535281 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84a791db69084487a71b65f282a4c83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a791db69084487a71b65f282a4c83e" member_type: VOTER last_known_addr { host: "127.18.90.62" port: 42513 } } peers { permanent_uuid: "5230806d085e444f81ba92040971a3dc" member_type: VOTER last_known_addr { host: "127.18.90.61" port: 45363 } } peers { permanent_uuid: "27eb45ec2df94e2a9e94fef185340b59" member_type: VOTER last_known_addr { host: "127.18.90.60" port: 37513 } } }
I20260806 16:16:45.535411 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:16:45.537403 9564 catalog_manager.cc:1347] Generated new cluster ID: d4c07250496a49b5b32da4ab186a099f
I20260806 16:16:45.537490 9564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:16:45.535952 9582 mvcc.cc:205] Tried to move back new op lower bound from 7315591190552231936 to 7315591190417383424. Current Snapshot: MvccSnapshot[applied={T|T < 7315591190552231936}]
I20260806 16:16:45.560464 9564 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:16:45.561791 9564 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:16:45.579855 9564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84a791db69084487a71b65f282a4c83e: Generated new TSK 0
I20260806 16:16:45.580806 9564 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:16:45.616317 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.612779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.18.90.62:42513)
0806 16:16:45.612920 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.612925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.614771 (+ 1846us) server_negotiation.cc:407] Connection header received
0806 16:16:45.615232 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.615236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.615317 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.615371 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.616071 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44732: BlockingRecv error: recv got EOF from 127.0.0.1:44732 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 16:16:45.617436 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.613574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.18.90.61:45363)
0806 16:16:45.613716 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.613721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.615149 (+ 1428us) server_negotiation.cc:407] Connection header received
0806 16:16:45.616673 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.616676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.616730 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.616781 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.617319 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39832: BlockingRecv error: recv got EOF from 127.0.0.1:39832 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 16:16:45.618960 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.614154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.18.90.60:37513)
0806 16:16:45.614301 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.614305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.617996 (+ 3691us) server_negotiation.cc:407] Connection header received
0806 16:16:45.618226 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.618231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.618296 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.618347 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.618841 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54598: BlockingRecv error: recv got EOF from 127.0.0.1:54598 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 16:16:45.623347 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.621756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.18.90.61:45363)
0806 16:16:45.621857 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.621861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.622390 (+ 529us) server_negotiation.cc:407] Connection header received
0806 16:16:45.622604 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.622607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.622684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.622736 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.623234 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39848: BlockingRecv error: recv got EOF from 127.0.0.1:39848 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 16:16:45.628468 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.626919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.18.90.61:45363)
0806 16:16:45.627024 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.627028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.627548 (+ 520us) server_negotiation.cc:407] Connection header received
0806 16:16:45.627727 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.627730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.627780 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.627830 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.628350 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39850: BlockingRecv error: recv got EOF from 127.0.0.1:39850 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 16:16:45.634124 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.632577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.18.90.61:45363)
0806 16:16:45.632692 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.632696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.633193 (+ 497us) server_negotiation.cc:407] Connection header received
0806 16:16:45.633388 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.633391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.633442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.633491 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.634023 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39866: BlockingRecv error: recv got EOF from 127.0.0.1:39866 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 16:16:45.646075 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.644522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.18.90.61:45363)
0806 16:16:45.644651 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.644655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.645137 (+ 482us) server_negotiation.cc:407] Connection header received
0806 16:16:45.645347 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.645350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.645407 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.645459 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.645967 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39868: BlockingRecv error: recv got EOF from 127.0.0.1:39868 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 16:16:45.658345 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.656690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.18.90.61:45363)
0806 16:16:45.656810 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.656814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.657377 (+ 563us) server_negotiation.cc:407] Connection header received
0806 16:16:45.657566 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.657569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.657621 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.657671 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.658220 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39874: BlockingRecv error: recv got EOF from 127.0.0.1:39874 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 16:16:45.678086 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.676847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.18.90.61:45363)
0806 16:16:45.677008 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.677013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.677134 (+ 121us) server_negotiation.cc:407] Connection header received
0806 16:16:45.677311 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.677315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.677383 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.677496 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.677962 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39886: BlockingRecv error: recv got EOF from 127.0.0.1:39886 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 16:16:45.713398 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.712175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.18.90.61:45363)
0806 16:16:45.712381 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.712386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.712585 (+ 199us) server_negotiation.cc:407] Connection header received
0806 16:16:45.712689 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.712691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.712747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.712820 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.713271 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39898: BlockingRecv error: recv got EOF from 127.0.0.1:39898 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260806 16:16:45.780959 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.779618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.18.90.61:45363)
0806 16:16:45.779789 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.779801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.779946 (+ 145us) server_negotiation.cc:407] Connection header received
0806 16:16:45.780129 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.780161 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.780217 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.780347 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.780842 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39912: BlockingRecv error: recv got EOF from 127.0.0.1:39912 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 16:16:45.912657 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:45.911342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.18.90.61:45363)
0806 16:16:45.911503 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:45.911508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:45.911664 (+ 156us) server_negotiation.cc:407] Connection header received
0806 16:16:45.911854 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:45.911857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:45.911905 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:45.911998 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:45.912538 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39914: BlockingRecv error: recv got EOF from 127.0.0.1:39914 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 16:16:46.172169 9447 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:16:46.170884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.18.90.61:45363)
0806 16:16:46.171030 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:46.171037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:46.171166 (+ 129us) server_negotiation.cc:407] Connection header received
0806 16:16:46.171368 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:46.171371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:46.171434 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:46.171526 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:16:46.172020 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39928: BlockingRecv error: recv got EOF from 127.0.0.1:39928 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:16:46.445142 9551 catalog_manager.cc:1259] Loaded cluster ID: d4c07250496a49b5b32da4ab186a099f
I20260806 16:16:46.445252 9551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: loading cluster ID for follower catalog manager: success
I20260806 16:16:46.446934 9551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: acquiring CA information for follower catalog manager: success
I20260806 16:16:46.448086 9551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5230806d085e444f81ba92040971a3dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:16:46.511925 9578 catalog_manager.cc:1259] Loaded cluster ID: d4c07250496a49b5b32da4ab186a099f
I20260806 16:16:46.512008 9578 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: loading cluster ID for follower catalog manager: success
I20260806 16:16:46.513718 9578 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: acquiring CA information for follower catalog manager: success
I20260806 16:16:46.514971 9578 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27eb45ec2df94e2a9e94fef185340b59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:16:46.690075 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:46.686391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.18.90.61:45363)
0806 16:16:46.686631 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:46.686635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:46.686757 (+ 122us) server_negotiation.cc:407] Connection header received
0806 16:16:46.686885 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:46.686887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:46.687014 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:46.687111 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:46.687340 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.687856 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.688738 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.688963 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.689039 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:46.689312 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:46.689751 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:46.689840 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:46.689899 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 16:16:46.689949 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260806 16:16:46.697058 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:46.693250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.18.90.60:37513)
0806 16:16:46.693486 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:46.693490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:46.693550 (+ 60us) server_negotiation.cc:407] Connection header received
0806 16:16:46.693705 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:46.693707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:46.694055 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:46.694171 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:46.694377 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.694894 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.695665 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.695879 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.695974 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:46.696223 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:46.696729 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:46.696826 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:46.696884 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 16:16:46.696934 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260806 16:16:46.704761 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:46.701075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.18.90.60:37513)
0806 16:16:46.701210 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:46.701215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:46.701227 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:16:46.701351 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:46.701354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:46.701405 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:46.701472 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:46.701805 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.702522 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.703565 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.703885 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.703981 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:46.704215 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:46.704346 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:46.704414 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:46.704569 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 16:16:46.704627 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:16:46.704916 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:46.700452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.18.90.61:45363)
0806 16:16:46.700617 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:46.700622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:46.701039 (+ 417us) server_negotiation.cc:407] Connection header received
0806 16:16:46.701258 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:46.701262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:46.701335 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:46.701423 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:46.701647 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.702411 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.703493 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.703814 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.703933 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:46.704421 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:46.704506 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:46.704571 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:46.704711 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 16:16:46.704770 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:16:46.705560 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:16:46.700091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.18.90.62:42513)
0806 16:16:46.700353 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 16:16:46.700359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:16:46.700376 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:16:46.700478 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:16:46.700482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:16:46.700652 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:16:46.700813 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:16:46.701327 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.702054 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.703112 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:16:46.703472 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:16:46.703578 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:16:46.704533 (+ 955us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:16:46.704905 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:16:46.705001 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:16:46.705331 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 16:16:46.705399 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260806 16:16:46.708654 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 9293
I20260806 16:16:46.716692 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 9371
I20260806 16:16:46.723342 18792 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG62Umc/build/debug/bin/kudu with pid 9449
2026-08-06T16:16:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3128 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6188 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285716 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST